What does `-p` option do?

[jmjf]

Problem

Working to fix #294, I realized I cannot figure out what the -p, --properties option is supposed to do. That's probably a "me" problem. Can someone please explain so we can understand how to fix the option for oas2tson or remove it.

If the team can provide some feedback on intent and preferred direction in light of discussion on #230, I don't mind trying to put together code and a PR.

Background

In oas2json and oas2tson, the option is defined as:

  .option(
    '-p, --properties <string>',
    'Comma-separated list of properties to convert from the OpenAPI file'
  )

The option is not available for oas2ts or json2ts.

The examples use -p paths, like openapi-transformer-toolkit oas2json -i ./openapi.yml -o ./schemas -p paths.

For oas2json, adding -p paths outputs a paths directory with JSON Schema files for the paths. Based on the description, I'd expect -p User,Tag to output only User and Tag. But adding a comma separated list fails with the message, "Failed to convert non-object attribute, skipping." Based on the code, components.schemas is always added to the list of properties.

For oas2tson, as reverted, -p paths fails. According to comments in #224, pets/findByStatus becomes const pets-findByStatus because - is not valid character in a JavaScript identifier. The - is not a problem for oas2json because oas2json writes JSON Schema, not variables.

Note PR seems to be trying to implement path generation, which later spawned #230 mentioned above. There, is seems like the team isn't convinced the tool should support TSON path generation. If it shouldn't, should it support JSON path generation either?

The name pets-findByStatus is really the filename for the output file. For example, oas2json writes file names like pets-findByStatus.json and pets-{petId}.json. The name is generated by filenamify(name, { replacement: '-' }) in both oas2tson and oas2json.

Questions

• What is --properties supposed to do? What benefit does it bring?

  • Is it supposed to include major branches of the OpenAPI spec other than components.schemas?
  • Is the idea we could have headers or other things too?
  • Is this a property or is it a directory in an OpenAPI spec that uses multiple files and directories?
  • Are there branches of the OpenAPI spec that should not be allowed?

• Should oas2tson support -p paths? (How symmetric should oas2json and oas2tson be?)

• If so how should path consts be named?

  • pets-findByStatus isn't valid JavaScript.
  • URLs are case sensitive so pets/findByStatus and pets/FindByStatus are different URLs, unless you configure your server to be case insensitive.
  • Maybe pets_findByStatus and pets_FindByStatus? Those are valid, but not idiomatic, JS names. Given URL case sensitivity not idiomatic may be necessary.

• Or if not, then should we remove the -p option from oas2tson?

Looking at oas2tson vs. oas2json, It feels wrong that they're so different. Maybe too simplistic, but TSON is just JSON with no quotes around the member names and a const <name> = in front of it and as const on the end. Somewhere in Nodeland there must be a JavaScript code generator that can take an object and generate output suitable for including in a file. json2ts is basically what we need, but with the values instead of the types. Then something like oas2json -> JSON.parse() -> object2json + as const should work.

[simoneb]

@ilteoood I'd like to hear your thoughts on this

[ilteoood]

It should give you the ability to choose the fields of an openapi file to convert.

In the following example, paths is an entry in the yaml that you can choose to process.

I think it has been created to allow the user to define what needs to be converted: request objects, response objects, general definitions or query parameters.
Said so, it's not something we should throw away.

[jmjf]

Okay. So the intent isn't to generate only the named schema but more about classes of objects. Does anyone have thoughts on the naming challenges? `oas2json` can use names `oas2tson` can't. But making `oas2tson` names valid by forcing strict camel case makes valid OpenAPI specs break. It may need to use a separator instead. Thoughts on preferred approaches?

…

On Mon, Aug 12, 2024, 04:49 Matteo Pietro Dazzi ***@***.***> wrote: It should give you the ability to choose the fields of an openapi file to convert. In the following example, paths is an entry in the yaml that you can choose to process. Screenshot_2024-08-12-10-46-02-790_com.github.android-edit.jpg (view on web) <https://github.com/user-attachments/assets/29b91af1-6dab-45ff-87f8-410f5095efc3> I think it has been created to allow the user to define what needs to be converted: request objects, response objects, general definitions or query parameters. Said so, it's not something we should throw away. — Reply to this email directly, view it on GitHub <#305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASIV63V64YDMRI23CJMD4DZRBZIVAVCNFSM6AAAAABMLFBZBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBTGQYTSMJUHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ilteoood]

I don't have a strong feeling about it, however you want to approach it, the final outcome is the important matter

[jmjf]

I'll work on making paths work with the following, unless some sees problems with this approach. Probably this weekend. - replace / and { and } in names with _ - honor upper/lower case from the spec So, `pets/{petId}` get a file and const named `pets__petId`. Or is it safe to reduce the double _ to one _? Is there a risk of `pets/petId` or a similar collision? TODO (note to self): check OpenAPI for other special characters to consider for paths and the major branches of the model that might benefit from TSON consts. Focus on Fastify schema capabilities for now.

…

On Tue, Aug 13, 2024, 06:46 Matteo Pietro Dazzi ***@***.***> wrote: I don't have a strong feeling about it, however you want to approach it, the final outcome is the important matter — Reply to this email directly, view it on GitHub <#305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASIV65JK35KRH2S7OTWWDDZRHPXTAVCNFSM6AAAAABMLFBZBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBVHE2DGMBRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ilteoood]

Honestly speaking I would avoid replacing / and {} with something else, because you could create some collisions with other path naming that the user could have.
Since those symbols have a specific meaning for OAS, I would use that.

[jmjf]

The existing code for both oas2json and oas2tson replaces / in paths and other names from the spec with - to get a filename. That works for oas2json because pets-{petId} is valid filename. pets/{petId} adds a directory layer.

oas2tson needs to write valid JavaScript/TypeScript identifier names. /, -, { and } are not valid characters in an identifier name. For example

// invalid identifier because /, {, }
const pets/{petId} = {
   ...
} as const;

// invalid identifier because -
const pets-petId = {
   ...
} as const;

oas2tson needs to use either _ or $ as a separator in the identifier name to get valid identifiers.

The files oas2tson writes should match the identifier name to avoid something like import { pets_petId } from 'path/to/tson/pets-{petId}'. I think _ is less likely to be a filename problem than $ (environment variable trigger).

Does this reasoning make sense for oas2tson given the need to write valid JavaScript/TypeScript?

I propose to make the replacements affect oas2tson only and PR it so it works. Then we can decide if oas2json needs any changes (probably doesn't).

[ilteoood]

The problem of using _ as a separator relies on a collision if someone defines the paths "pets/{petId}" and "pets_petId".
On a REST standard perspective the second path doesn't make any sense, but technically speaking is something that can be defined.
The same could apply also for the dollar sign.

I see what's the issue here, we are trying to map an URL without char limitations to something that is limited like a TypeScript type...

[jmjf]

`pets/{petId}` could map to `pets__petId` or keep the trailing _ to reduce collision risk, but I acknowledge that doesn't solve the problem you raise. I'm open to ideas if someone sees a better solution.

…

On Tue, Aug 13, 2024, 13:20 Matteo Pietro Dazzi ***@***.***> wrote: The problem of using _ as a separator relies on a collision if someone defines the paths "pets/{petId}" and "pets_petId". On a REST standard perspective the second path doesn't make any sense, but technically speaking is something that can be defined. The same could apply also for the dollar sign. I see what's the issue here, we are trying to map an URL without char limitations to something that is limited like a TypeScript type... — Reply to this email directly, view it on GitHub <#305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASIV66I6EM3I7LKVGPCMITZRI57FAVCNFSM6AAAAABMLFBZBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWG42DOMJRGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jmjf]

I thought through this problem today to plan what I need to do with oas2tson to support -p paths. The naming challenges noted in previous comments make this problem complex. Some OpenAPI features cannot be supported in oas2tson without butchering OpenAPI names beyond recognition.

I plan to start coding tomorrow, but this may take a while. If anyone has suggestions or solutions for some of the problems or sees issues I missed, let me know.

TL;DR:

• Changes affect oas2tson only and must produce valid TypeScript object output.
• Only components, paths, and webhooks top level members need to work. oas2json doesn't support most other top level members. It does support info and tags, but they're low value, so nice if they work but not required.
• -p paths, not -p paths.parameters, etc. -- oas2json fails on the latter. (Assume same for webhooks.)
• components.links and components.examples are low value, so nice to have but not required.
• components.callbacks is not required because it uses URL strings as member names (example in discussion below). For the same reason, callbacks in paths and webhooks are not required.
• oas2tson will add directories, similar to oas2json, to avoid name collisions. I'll aim to move components.schemas contents up to avoid breaking existing imports.
• Use _ as a replacement for JavaScript-bad characters; trim leading/trailing, keep dupes

First thoughts

Running oas2json -p paths writes output to <target-directory>/components.schemas and <target-directory>/paths. oas2tson (and oas2ts) write all output to the target directory. With the example pet store schema:

• If I write components.schemas output and paths output to the same directory, I'll end up with Pet.tson and pet.tson in the same directory. That's a problem for Windows users (case insensitive file names).
• If I introduce a components.schemas directory, the new directory layer breaks existing imports. So, this choice is a breaking change.
• I could leave schemas where they are and introduce a paths directory, but that's asymmetric compared to oas2json. TBD if that makes sense.
• I could prefix all paths with Path, which would get Pet.tson and Path_pet.tson from the example. I'm not sure this is a better option than the directory or breaking the directory structure contract

What motivated these features?

Exporting paths and other parts of the spec besides components.schemas with oas2json -- #54

• Relates it to a feature added to openapi-schema-to-json-schema. oas2tson uses the same tool, so if oas2json doesn't support some part of the spec, consider it not required.

Exporting as TSON (oas2tson) -- #35

• The main motivation for oas2tson is supporting Fastify features and avoiding JSON imports. Don't rule out other use cases, of course.

What does oas2json do?

Does oas2json support the following top-level members of the spec?

npm run oas2json -- -p <member-name>

• openapi - no; "Failed to convert non-object attribute, skipping"
• info - exports contact and license only; seems low value
• jsonSchemaDialect - not in the example spec, but I'm guessing no, or if it did this isn't valuable
• servers - no; "Failed to convert non-object attribute, skipping"
• security - not in example spec, but I'm guessing no, or if it did it's low value -- defined in OpenAPI as a string
• tags - exports the name and description for each tag; seems low value
• externalDocs - no; "Failed to convert non-object attribute, skipping"

Does oas2json support the following lower level members of the spec?

Both oas2json and oas2tson always export components.schemas and work.

oas2tson requires JavaScript friendly names. The OpenAPI spec does not require JavaScript friendly names and requires names that aren't JavaScript friendly for some things. For example, the components.callbacks examples include 'http://notificationServer.com?transactionId={$request.body#/id}&email={$request.body#/email}' as a member name. That's fine if you wrap it in quotes, but oas2tson unquotes names, so it doesn't work for oas2tson. It may be that the main reason oas2tson has so few problems with components.* members is that its users are JS/TS folks who use JS/TS naming patterns in their components.schemas.

oas2json references $refs, but oas2tson embeds $refs. So, if I $ref: '#/components/examples/Foo' in components.schemas.Pet, the full definition of Foo will be embedded in Pet.tson, not referenced. API authors consuming the TSON cannot change myExample and affect Pet -- ignoring the fact oas2tson declares everything as const so it's immutable.

I added sections to the example spec to support testing what oas2json and oas2tson do with different -p values. In the list below, the first result is oas2json.

• components.securitySchemes - no, "Failed to convert non-object attribute, skipping"; oas2tson fails too
• components.requestBodies - yes; works with oas2tson after renaming the Pet request body to avoid a collision with the Pet schema
• components.parameters - yes; works with oas2tson; note possible collisions with schemas for object-type parameters or compound parameter schemas
• components.examples - yes; works with oas2tson; seems low value as a standalone; nice to have
• components.responses - yes; works with oas2tson
• components.pathItems - yes; works with oas2tson; I'm unsure to write these and @redocly/cli lint is complaining about components.pathItems, so may have it wrong
• components.headers - yes; works oas2tson after removing the - from the header name I tested
  • Demonstration of oas2tson's current requirement for JS friendly names.

components can include the following. I didn't try to test these cases.

• components.links - This makes me think links are nice to have but not worth extra effort.
• components.callbacks - Callbacks involve full and parameterized URLs as names, so I won't invest extra effort in callbacks. See the example cited above. oas2json may produce valid output, oas2tson would need to brutalize the name.

Callbacks may appear in paths and webhooks. Do not expect oas2tson to support paths and webhooks with callbacks.

Decisions

• Only components, paths, and webhooks top-level members are in scope.
• For paths, exporting only paths.parameters or only paths.get, etc., doesn't make sense. If they work, great, but I'm not going to care if they don't. If you care about separate parameters objects, you should have defined them in components anyway. oas2json fails given -p paths.parameters, supporting this choice.
• components.securitySchemes, components.links and components.callbacks are out of scope for reasons stated or implied above.
• For TSON const names and filenames, convert all characters that are not valid in JavaScript identifiers to _. Remove leading and trailing _, but keep duplicates.
  • Examples: /api/pet/{petId} -> api_pet__petId; /pet/findByStatus -> pet_findByStatus; Content-Type -> Content_Type
  • oas2json writes /pet/{petId} to pet-{petId}.json with "title": "/pet/{petId}", "$id": "pet-petId.json"
  • Assumption: oas2tson should follow a similar pattern but with filename and $id matching const name.
• Introduce directories in oas2tson output. If practical, move files in components.schemas to the output directory to avoid breaking existing imports.
  • Assumption: That asymmetry with oas2json and within the output directory itself is acceptable in the interest of being non-breaking.

[jmjf]

@ilteoood; @simoneb

As I finished that long comment outlining the problem, I had a crazy thought. What's your opinion on this?

oas2tson should take the JSON from oas2json and build const <name> = <JSON-text> as const;, where:

• <name> is a JavaScript friendly name (use the pattern described above with _ replacements, trimming, etc.).
• <JSON-text> is the JSON file content produced by oas2json.

That would get output something like the following for the /user path after applying prettier. Note the content section uses JS-unfriendly names, but is still valid JS/TS because they're wrapped in quotes.

const user = {
  post: {
    tags: ['user'],
    summary: 'Create user',
    description: 'This can only be done by the logged in user.',
    operationId: 'createUser',
    requestBody: {
      description: 'Created user object',
      content: {
        'application/json': {
          schema: {
            $ref: 'User.json'
          }
        },
        'application/xml': {
          schema: {
            $ref: 'User.json'
          }
        },
        'application/x-www-form-urlencoded': {
          schema: {
            $ref: 'User.json'
          }
        }
      }
    },
    responses: {
      default: {
        description: 'successful operation',
        content: {
          'application/json': {
            schema: {
              $ref: 'User.json'
            }
          },
          'application/xml': {
            schema: {
              $ref: 'User.json'
            }
          }
        }
      }
    }
  },
  title: '/user',
  $id: 'user.json'
} as const;`

So, oas2tson becomes something like:

• Run oas2json to temp directory.
• Recurse temp directory
  • If a directory, create a corresponding directory in the target directory
  • If a file
    • Make file name JavaScript friendly (newName).
    • Build a string as const <newName> = <JSON-text> as const;
    • Run prettier on string
    • Write file
• Move contents of components.schemas into target directory and remove components.schemas

Later

Plan

• Write json2tson to write TSON files with JS-friendly names for consts and file names.
• oas2tson -> oas2json + json2tson + copy components.schemas to target directory so imports don't break.

Even later

It's more complex than that because oas2tson wants to dereference $refs.

In oas2tson.ts, line 54 has a comment that suggests the code supports expanding $refs where the ref is to components/schemas only. If a path uses $refs to components/headers, components/parameters, etc., the original oas2tson design does not support it. The same is true for schemas that reference components/examples.

This effort will involve either expanding how oas2tson handles dereferencing so paths and references to components/examples will work or deciding that isn't practical.

More to come. At this point, I believe I'll need a week or three to work through the details.

[jmjf]

@ilteoood, @simoneb

Why does oas2json strip the path information from refs?

npm run oas2json -- -p paths

Look at example/output/json/paths/pet.json and note the $refs to Pet are all Pet.json without #/components/schemas/. On case-insensitive file systems, the pet route is referencing itself. On case sensitive file systems, it's "file not found" unless you also have a Pets path, which isn't what you really want. If I $ref headers, parameters, and other components, in the path, same problem.

The same thing happens in the other subdirectories. For example, add to the following to components in the example schema.

  callbacks:
    userCreatedCallback:
      'https://notify.example.com?event=UserCreated':
        post:
          operationId: notifyUserCreatedCB
          summary: notify on user create
          requestBody:
            description: Callback payload
            content:
              'application/json':
                schema:
                  $ref: '#/components/schemas/User'
          responses:
            '200':
              description: Callback succeeded
            '4XX':
              description: Callback failed

and run with -p components.callbacks. In the JSON, the $ref to User has no path, so file not found if you try to find the corresponding file.

The stripping happens in oas2json.ts lines 51-54

https://github.com/nearform/openapi-transformer-toolkit/blob/aced54d3a61c326795b2a050adf9e8094ce0c336/src/commands/oas2json.ts#L51C1-L54C7

I could change that to get a path like ../components.schemas/Pet.json. But I'd like to know why it's the way it is to be sure I don't break something else or do something that breaks the intent of the tool. I'm assuming one wouldn't write something like $ref: '#/components/schemas/Pet/status.

I think that change would require changing json2ts to strip the paths because it supports components.schemas only. (Line 27 passes undefined to oas2json for propertiesToExport, so will only get components.schemas.)

Digging through this, I'm back to the idea that oas2tson is just oas2json plus the processJSON function to dereference and wrap the export const... part around the JSON.

[ilteoood]

It's made in this way to be easily imported by ajv, which doesn't need a path to import the file but just its title.

[jmjf]

Thank you. Another TIL -- why oas2tson duplicates code from other parts of the tool. Any day I can say, "TIL..." several times is a good day. 😄

I think that pushes me toward prefixing names for everything but schemas to avoid collisions (in the current example, something like Pet.tson (schema), RequestBodies_Pet.tson, Paths_pet.tson) instead of using directories. More to come.