Ionic Encrypted Search (IonES) system is a lightweight and scalable multi-user searchable encryption solution that enables users to search and retrieve protected documents in a provably secure way. IonES queries involve users and services participating in secure multi-party computations to eliminate a single point of failure.
Why? A Founder’s Perspective
We began work on Ionic Machina, almost eight years ago, hell bent on the mission to enable machine-scale trust in a machine-scale world we began with how to decouple the security and privacy of data from that of the infrastructure. As Machina began to take shape, our early clients consistently proposed a simple, yet provocative, thought experiment:
“One day, when Ionic has succeeded you will have accomplished two things:
- Ionic will have encrypted all of my data and made the use thereafter easy to manage at scale.
- Ionic will have encrypted all of my data and broken a core business function: Search.
What will you do then?”
While this thought experiment introduced a tremendous future challenge, it did not slow our clients or ourselves down. Our earliest clients were willing to accept some loss of business function in exchange for the significant increases in security and accountability that Machina provides.
There was enough data to go around, which meant initially focusing on use cases where search was not a critical component of the business process.
However, as we advanced deeper into the global data estates of our clients, we realized that the search challenge would come to a head sooner rather than later. With our minds open, we devised a parallel path strategy with a long arc of success as the foundation: review any solution that could solve this problem with the intent to partner with the creators, while independently conducting our own research into the subject matter in collaboration with academia and industry.
As we introduced many in-market offerings to our clients as potential solutions, multiple common themes as to why they would not suffice in practice became apparent. IonES is the result of over four years of dedicated collaboration with our industry partners, academic partners, and a tremendously talented team of Ions.
Below, and contained within the various whitepapers referenced, is the outcome of all of these efforts. IonES is a fully-functional development and service framework to provide an end-to-end private and encrypted modern search service that allows for arbitrary encryption algorithms, such as AES GCM, to be utilized for the data protection scheme.
The framework enables many of the practical needs of a modern search and retrieval platform while remaining provider-agnostic.
We would like to thank Professors David Cash (University of Chicago) and Dan Boneh (Stanford University) for their many thoughtful cycles reviewing every minute aspect of the IonES framework over the years. Thank you to our clients and partners for your cycles distilling theory into practice with practical application as the driving force.
Last, and certainly not least, thank you to all of the incredible Ions who dreamed, failed, and persisted through to this amazing outcome together!
Should you feel compelled to collaborate with Ionic on the topic of IonES, we are hiring! Please reach out to us here.
All the best,
Founder, Ionic Security
by Jonathan Burns, PhD (Head, Cryptography and Mathematics – Ionic Security)
Data in use is the most volatile form of data; protecting it depends upon a host of security and privacy measures designed to limit access and minimize the time the data spends in cleartext.
Large enterprises maintain private repositories in hopes of containing sensitive data like intellectual property or personal client data affected by changing regulations. However, private repositories incur costs for hardware, software, technicians, auditors, and specialists to prevent data breaches and potential loss. In 2018, Forbes reported that there are 2.5 quintillion bytes of data created every day, and it’s no surprise that an organization’s failure to properly predict skyrocketing storage needs results in excessive infrastructure expenditures or in hitting capacity limitations that lead to data loss.1
To avoid the complicated and expensive endeavor of large on-premises repositories, professionals are turning to the cloud. Cloud storage providers promise a simple and inexpensive alternative to on-premises hosting, but require an enterprise to “trust the cloud.” Sensitive data can be protected in a public context using a combination of strong encryption schemes, key management systems, and access control policies. However, protecting a document generally removes the ability to search over it. Data in use extends to decrypting files to perform common business tasks such as e-discovery.
Advanced cryptographic techniques like homomorphic encryption (HE) promise to deliver search results while the files themselves remain encrypted, but that promise has yet to materialize in a way that maximizes the benefits of cloud-delivered storage architectures. HE allows evaluation of a secure document without decrypting it, but modern HE schemes are either weaker than standard symmetric ciphers, like AES, are computationally prohibitive, or significantly constrain the delivery architecture of the search service.
Even if HE technology improves to the point where ciphertext can be processed as quickly as plaintext, it is still impractical to perform global searches across individual files in a large repository for every search request. Further, HE alone does not solve for the privacy needs that co-mingling of data with different levels of sensitivity in a common repository requires. Lastly, practical search services require capabilities such as arbitrary and dynamic insertions and deletions, as well as the need to merge indexes created in different locations at different points in time. Capabilities such as these are non-trivial with HE schemes.
IonES provides solutions for these practical challenges via the search of secure indexes, safeguarded through multi-party computations. After reading about the design of key processes and available implementation resources, interested parties can sign up to participate in our private beta. IonES will transition into a publicly available offering in the not-too-distant future.
The IonES system can be boiled down to two processes:
- Query Construction: Clients participate in a multiparty computation with a trusted stateless service known as the EC-OPRF (Elliptic Curve Oblivious Pseudo-Random Function) provider to generate secure tokens for INSERT, SEARCH, and DELETE queries. EC-OPRF providers create a cryptographic compartmentalization of data within an index service.
- Query Execution: Clients submit their secure queries to the index service, which for the sake of argument, is not trusted. The index service is able to process the queries and return results, while remaining provably oblivious to both the query and results.
A single index service can process queries and deliver results for an unlimited number of EC-OPRF providers without compromising the privacy or security guarantees of any EC-OPRF compartment.
Unlike most searchable encryption schemes, IonES relies upon generic cryptographic primitives – e.g., block ciphers, hash functions, pseudo-random functions, etc. – and can be adapted accordingly to meet many practical constraints such as regulatory or performance requirements.
A more thorough design review of the IonES framework can be obtained by reading Securing the Cloud with Client-Side Encryption: Benefits and Challenges by Professor David Cash at the University of Chicago.
During the query construction process, a client interacts with a third-party service to use the EC-OPRF protocol (introduced in 2017 by Ionic Research) to compute a secure search token from the client’s search term and a third-party’s private key. The EC-OPRF protocol provides Shannon perfect secrecy against the third-party learning the client’s search term, and the third-party’s private key is protected from the client by the hardness of the elliptic curve discrete logarithm problem. In this way, a client cannot construct a query independently from the third-party EC-OPRF service.
The EC-OPRF service can manage multiple keys to regulate query access control. For example, queries for the search term “cat” which are signed using the private keys for “public data”, “confidential data”, and “restricted data” will produce distinct secure search tokens. Users provisioned under the “confidential data” EC-OPRF provider will not be able to query “restricted data” keyed results stored in the the same index service. Further, the index service is unable to distinguish between queries generated from different keys. The computations performed by the EC-OPRF service are extremely lightweight and highly parallelizable, comparable to creating TLS connections.
The IonES system supports query execution over a Hash and Inverted Index Table (HiiT), a proprietary, secure, inverted index data structure. The HiiT index supports insertion, deletion, and boolean search queries over ranked entries. Each set of document indexes associated with a search term are encrypted using a distinct symmetric key that is generated from its corresponding secure search token, and accessed via a hash value that is also generated from the secure search token. This means that a client’s plaintext search terms are never disclosed to the index service. Likewise, the indexes of the protected documents can be encrypted client-side before insertion into the HiiT, so that index service stays opaque as it computes the search results.
The IonES beta version includes the following client-side and server-side resources.
Installation and support documentation is provided for all of the following applications.
- Golang (coming soon)
- Sample Applications:
- CLI Tool – Command line execution of SEARCH and INSERT / DELETE queries for documents into the index.
- Folder Watcher – Once a file is moved to an INSERT watched folder, the application secures and uploads the document to AWS S3, then inserts the secure tokens into the index. Moving a file into a DELETE watched folder removes the document from S3 and deletes the secure tokens from the index.
- Web Search Interface –Search, download and decrypt encrypted data, and delete tokens through a web browser.
- Web Search APIs – Access to the APIs that power the Web Search Interface.
Installation, support, and API reference documentation is provided for all of the following applications.
- EC-OPRF Services: VMWare OVA, Amazon AMI
- Index Services: VMWare OVA, Amazon AMI
By offering a fast, trusted alternative to homomorphic encryption, IonES removes another barrier for enterprises and application developers interested in adopting cloud technologies for their most sensitive data.
- Forbes, How Much Data Do We Create Every Day? The Mind-Blowing Stats Everyone Should Read, Bernard Marr, 21 May 2018
- EC-OPRF (Ionic Research)
- Securing the Cloud with Client-Side Encryption: Benefits and Challenges (Professor David Cash, University of Chicago)