Persisted queries

docs/Modern-GraphQLInRelay.md

@@ -144,7 +144,33 @@ Will cause a generated file to appear in `./__generated__/MyComponent.graphql`,
 with both runtime artifacts (which help to read and write from the Relay Store)
 and [Flow types](https://flow.org/) to help you write type-safe code.
 
-The Relay Compiler is responsible for generating code as part of a build step which, at runtime, can be used statically. By building the query ahead of time, the client's JS runtime is not responsible for generating a query string, and fields that are duplicated in the query can be merged during the build step, to improve parsing efficiency. If you have the ability to persist queries to your server, the compiler's code generation process provides a convenient time to convert a query or mutation's text into a unique identifier, which can greatly reduce the upload bytes required in some applications.
+The Relay Compiler is responsible for generating code as part of a build step which, at runtime, can be used statically. By building the query ahead of time, the client's JS runtime is not responsible for generating a query string, and fields that are duplicated in the query can be merged during the build step, to improve parsing efficiency.
+
+### Persisting queries
+The Relay Compiler can convert a query or mutation's text into a unique identifier during compilation. This can greatly reduce the upload bytes required in some applications.
+Using a unique identifier in place of the query text means you have a well-known list of queries that the client can send instead of any
+arbitrary query. This technique of allowing only known queries to be run on the server is called query whitelisting. Whitelisting prevents malicious attacks
+such as:
+
+* an internal dos attack where the query depth is high and exploits circular relationships in the schema
+* a full schema introspection which might not be desirable
+* a harmful pagination query that fetches a million objects
+
+For more information on whitelisting and securing your graphql api from attacks, please refer to this [excellent blog by Max Stoiber](https://dev-blog.apollodata.com/securing-your-graphql-api-from-malicious-queries-16130a324a6b).
+
+The Relay Compiler can persist your queries with the `--persist` flag:
+
+```js
+"scripts": {
+  "relay": "relay-compiler --src ./src --schema ./schema.graphql --persist"
+}
+```
+
+This will create a matching `./__generated__/MyComponent.queryMap.json` containing the query id and the operation text of the query in the same directory.
+The Relay Compiler aggregates all the generated `*.queryMap.json` files into a single complete query map file at `./src/complete.queryMap.json`. You can then use this complete
+json file in your server side to map query ids to operation text.
+
+For more details, refer to the [Persisted Queries section](./persisted-queries.html).
 
 ### Set up relay-compiler
 
@@ -222,6 +248,11 @@ This would produce three generated files, and two `__generated__` directories:
 * `src/Components/__generated__/DictionaryComponent_definition.graphql.js`
 * `src/Queries/__generated__/DictionaryQuery.graphql.js`
 
+If you use `--persist`, then an extra query map json file will also be generated:
+
+* `src/Queries/__generated__/DictionaryQuery.queryMap.json`
+
+Only one query map json file is generated in this instance because only concrete queries can be persisted. Fragments are not persisted.
 
 ### Importing generated definitions
 

docs/Modern-PersistedQueries.md

@@ -0,0 +1,164 @@
+---
+id: persisted-queries
+title: Persisted Queries
+---
+
+The relay compiler supports persisted queries which is useful because:
+
+* the client operation text becomes just an md5 hash which is usually shorter than the real
+query string. This saves upload bytes from the client to the server.
+
+* the server can now whitelist queries which improves security by restricting the operations
+that can be run from the client.
+
+## Usage on the client
+
+### The `--persist` flag
+In your `npm` script in `package.json`, run the relay compiler using the `--persist` flag:
+
+```js
+"scripts": {
+  "relay": "relay-compiler --src ./src --schema ./schema.graphql --persist"
+}
+```
+
+The `--persist` flag does 3 things:
+
+1. It converts all query and mutation operation texts to md5 hashes.
+
+    For example without `--persist`, a generated `ConcreteRequest` might look like below:
+
+    ```js
+    const node/*: ConcreteRequest*/ = (function(){
+    //... excluded for brevity
+    return {
+      "kind": "Request",
+      "operationKind": "query",
+      "name": "TodoItemRefetchQuery",
+      "id": null, // NOTE: id is null
+      "text": "query TodoItemRefetchQuery(\n  $itemID: ID!\n) {\n  node(id: $itemID) {\n    ...TodoItem_item_2FOrhs\n  }\n}\n\nfragment TodoItem_item_2FOrhs on Todo {\n    text\n    isComplete\n}\n",
+      //... excluded for brevity
+    };
+    })();
+    ```
+
+    With `--persist` this becomes:
+
+    ```js
+    const node/*: ConcreteRequest*/ = (function(){
+    //... excluded for brevity
+    return {
+      "kind": "Request",
+      "operationKind": "query",
+      "name": "TodoItemRefetchQuery",
+      "id": "3be4abb81fa595e25eb725b2c6a87508", // NOTE: id is now an md5 hash of the query text
+      "text": null, // NOTE: text is null now
+      //... excluded for brevity
+    };
+    })();
+    ```
+
+2. It generates a matching `.queryMap.json` file containing a map of the id and the operation text in the same `__generated__`
+directory as the `.graphql.js` file. In the example above, the `__generated__` directory will have these files:
+
+    * `./__generated__/TodoItemRefetchQuery.graphql.js`
+    * `./__generated__/TodoItemRefetchQuery.queryMap.json`
+
+    The `.queryMap.json` file looks something like this:
+
+    ```json
+    {
+      "3be4abb81fa595e25eb725b2c6a87508": "query TodoItemRefetchQuery(\n  $itemID: ID!\n) {\n  node(id: $itemID) {\n    ...TodoItem_item_2FOrhs\n  }\n}\n\nfragment TodoItem_item_2FOrhs on Todo {\n    text\n    isComplete\n}\n"
+    }
+    ```
+
+3. It also generates a complete query map file at `[your_src_dir]/complete.queryMap.json`. This file contains all the query ids
+and their operation texts. You can specify a custom file path for this file by using the `--persist-output` option:
+
+```js
+"scripts": {
+  "relay": "relay-compiler --src ./src --schema ./schema.graphql --persist --persist-output ./src/queryMaps/queryMap.desktop.queryMap.json"
+}
+```
+
+The example above writes the complete query map file to `./src/queryMaps/queryMap.desktop.queryMap.json`. You need to ensure all the directories
+leading to the `queryMap.json` file exist. Also note that the file extension has to be `.json`.
+
+### Network layer changes
+You'll need to modify your network layer fetch implementation to pass a documentId parameter in the POST body instead of a query parameter:
+
+```js
+function fetchQuery(operation, variables,) {
+  return fetch('/graphql', {
+    method: 'POST',
+    headers: {
+      'content-type': 'application/json'
+    },
+    body: JSON.stringify({
+      documentId: operation.id, // NOTE: pass md5 hash to the server
+      // query: operation.text, // this is now obsolete because text is null
+      variables,
+    }),
+  }).then(response => {
+    return response.json();
+  });
+}
+```
+
+## Usage on the server
+On the server, you'll need to map the query id in the POST body to the real operation text. You can utilise the
+`complete.queryMap.json` file to do this so you'll need a copy of it on your server.
+
+For universal applications where the client and server code are in one project, this is not an issue since you can place
+the query map file in a common location accessible to both the client and the server.
+
+### Compile time push
+For applications where the client and server projects are separate, one option is to have an additional npm run script
+to push the query map at compile time to a location accessible by your server:
+
+```js
+"scripts": {
+  "push-queries": "node ./pushQueries.js",
+  "relay": "relay-compiler --src ./src --schema ./schema.graphql --persist && npm run push-queries"
+}
+```
+
+Some possibilities of what you can do in `./pushQueries.js`:
+
+* `git push` to your server repo
+
+* save the query maps to a database
+
+### Run time push
+A second more complex option is to push your query maps to the server at runtime, without the server knowing the query ids at the start.
+The client optimistically sends a query id to the server, which does not have the query map. The server then in turn requests
+for the full query text from the client so it can cache the query map for subsequent requests. This is a more complex approach
+requiring the client and server to interact to exchange the query maps.
+
+### Simple server example
+Once your server has access to the query map, you can perform the mapping. The solution varies depending on the server and
+database technologies you use, so we'll just cover the most common and basic example here.
+
+If you use `express-graphql` and have access to the query map file, you can import the `complete.queryMap.json` file directly and
+perform the matching using the `matchQueryMiddleware` from [relay-compiler-plus](https://github.com/yusinto/relay-compiler-plus).
+
+```js
+import Express from 'express';
+import expressGraphql from 'express-graphql';
+import {matchQueryMiddleware} from 'relay-compiler-plus';
+import queryMapJson from './complete.queryMap.json';
+
+const app = Express();
+
+app.use('/graphql',
+  matchQueryMiddleware(queryMapJson),
+  expressGraphl({schema}));
+```
+
+## Using `--persist` and `--watch`
+It is possible to continuously generate the query map files by using the `--persist` and `--watch` options simultaneously.
+This only makes sense for universal applications i.e. if your client and server code are in a single project
+and you run them both together on localhost during development. Furthermore, in order for the server to pick up changes
+to the `queryMap.json`, you'll need to have server side hot-reloading set up. The details on how to set this up
+is out of the scope of this document.
+

packages/relay-compiler/bin/RelayCompilerBin.js

@@ -22,6 +22,7 @@ const {
 const RelaySourceModuleParser = require('../core/RelaySourceModuleParser');
 const RelayFileWriter = require('../codegen/RelayFileWriter');
 const RelayIRTransforms = require('../core/RelayIRTransforms');
+const persistQuery = require('../codegen/persistQuery');
 const RelayLanguagePluginJavaScript = require('../language/javascript/RelayLanguagePluginJavaScript');
 
 const fs = require('fs');
@@ -139,6 +140,8 @@ async function run(options: {
   watch?: ?boolean,
   validate: boolean,
   quiet: boolean,
+  persist: boolean,
+  'persist-output': string,
   noFutureProofEnums: boolean,
   language: string,
   artifactDirectory: ?string,
@@ -149,7 +152,26 @@ async function run(options: {
   }
   const srcDir = path.resolve(process.cwd(), options.src);
   if (!fs.existsSync(srcDir)) {
-    throw new Error(`--source path does not exist: ${srcDir}.`);
+    throw new Error(`--src path does not exist: ${srcDir}.`);
+  }
+
+  const persist = options.persist;
+  let persistOutput = options['persist-output'];
+  if (persistOutput) {
+    persistOutput = path.resolve(process.cwd(), persistOutput);
+    const persistOutputDir = path.dirname(persistOutput);
+    if (!fs.existsSync(persistOutputDir)) {
+      throw new Error(
+        `--persist-output path does not exist: ${persistOutputDir}.`,
+      );
+    }
+
+    const persistOutputFileExtension = path.extname(persistOutput);
+    if (persistOutputFileExtension !== '.json') {
+      throw new Error(
+        `--persist-output must be a path to a .json file: ${persistOutput}.`,
+      );
+    }
   }
   if (options.watch && !options.watchman) {
     throw new Error('Watchman is required to watch for changes.');
@@ -244,9 +266,12 @@ Ensure that one such file exists in ${srcDir} or its parents.
         languagePlugin,
         options.noFutureProofEnums,
         artifactDirectory,
+        persist,
+        persistOutput,
       ),
       isGeneratedFile: (filePath: string) =>
-        filePath.endsWith('.graphql.' + outputExtension) &&
+        (filePath.endsWith('.graphql.' + outputExtension) ||
+          filePath.endsWith('.queryMap.json')) &&
         filePath.includes(generatedDirectoryName),
       parser: sourceParserName,
       baseParsers: ['graphql'],
@@ -281,6 +306,8 @@ function getRelayFileWriter(
   languagePlugin: PluginInterface,
   noFutureProofEnums: boolean,
   outputDir?: ?string,
+  persist: boolean,
+  persistOutput: string,
 ) {
   return ({
     onlyValidate,
@@ -305,6 +332,8 @@ function getRelayFileWriter(
         optionalInputFieldsForFlow: [],
         schemaExtensions,
         useHaste: false,
+        persistQuery: persist ? persistQuery : undefined,
+        persistOutput,
         noFutureProofEnums,
         extension: languagePlugin.outputExtension,
         typeGenerator: languagePlugin.typeGenerator,
@@ -426,6 +455,14 @@ const argv = yargs
       type: 'boolean',
       default: false,
     },
+    persist: {
+      describe: 'Use an md5 hash as query id to replace operation text',
+      type: 'boolean',
+    },
+    'persist-output': {
+      describe:
+        'The json filepath where the complete query map file will be written to',
+    },
     noFutureProofEnums: {
       describe:
         'This option controls whether or not a catch-all entry is added to enum type definitions ' +

packages/relay-compiler/bin/__tests__/RelayCompilerBin-test.js

@@ -0,0 +1,89 @@
+/**
+ * Copyright (c) Facebook, Inc. and its affiliates.
+ *
+ * This source code is licensed under the MIT license found in the
+ * LICENSE file in the root directory of this source tree.
+ *
+ * @format
+ * @emails oncall+relay
+ */
+
+jest.mock('graphql-compiler');
+jest.mock('yargs');
+jest.mock('@babel/polyfill');
+
+require('graphql-compiler');
+const yargs = require('yargs');
+
+describe('RelayCompilerBin', () => {
+  let originalProcessExit;
+  let originalConsoleError;
+
+  const mockCliArguments = (schema = './', src = './', persistOutput = './') => {
+    yargs.usage.mockImplementation(() => ({
+        options: () => ({
+          help: () => ({
+            argv: {schema, src, 'persist-output': persistOutput},
+          }),
+        }),
+      })
+    );
+  };
+
+  beforeEach(() => {
+    jest.resetModules();
+    originalProcessExit = process.exit;
+    originalConsoleError = console.error;
+    process.exit = jest.fn();
+    console.error = jest.fn();
+  });
+
+  afterEach(() => {
+    process.exit = originalProcessExit;
+    console.error = originalConsoleError;
+  });
+
+  test('should throw error when schema path does not exist', async () => {
+    mockCliArguments('./some/path/schema.graphql');
+
+    await require('../RelayCompilerBin');
+
+    expect(console.error).toBeCalled();
+    expect(console.error.mock.calls[0][0]).toEqual(expect.stringContaining('Error: --schema path does not exist:'));
+    expect(process.exit).toBeCalled();
+    expect(process.exit.mock.calls[0][0]).toEqual(1);
+  });
+
+  test('should throw error when src path does not exist', async () => {
+    mockCliArguments('./', './some/path/src');
+
+    await require('../RelayCompilerBin');
+
+    expect(console.error).toBeCalled();
+    expect(console.error.mock.calls[0][0]).toEqual(expect.stringContaining('Error: --src path does not exist:'));
+    expect(process.exit).toBeCalled();
+    expect(process.exit.mock.calls[0][0]).toEqual(1);
+  });
+
+  test('should throw error when persist-output path does not exist', async () => {
+    mockCliArguments('./', './', 'some/path/complete.queryMap.json');
+
+    await require('../RelayCompilerBin');
+
+    expect(console.error).toBeCalled();
+    expect(console.error.mock.calls[0][0]).toEqual(expect.stringContaining('Error: --persist-output path does not exist:'));
+    expect(process.exit).toBeCalled();
+    expect(process.exit.mock.calls[0][0]).toEqual(1);
+  });
+
+  test('should throw error when persist-output path does not end with a .json extension', async () => {
+    mockCliArguments('./', './', './queryMap.graphql.js');
+
+    await require('../RelayCompilerBin');
+
+    expect(console.error).toBeCalled();
+    expect(console.error.mock.calls[0][0]).toEqual(expect.stringContaining('Error: --persist-output must be a path to a .json file'));
+    expect(process.exit).toBeCalled();
+    expect(process.exit.mock.calls[0][0]).toEqual(1);
+  });
+});