Removing the dictionary?

[akijakya]

Background

The dictionary in the OCSF schema (specifically, dictionary.json) is the authoritative catalog of all standardized attributes used across OCSF event classes and objects. Its main purposes are:

• Standardization: It defines each attribute’s properties (such as name, data type, description, and, where applicable, enumerated values or references), ensuring that attributes are used consistently throughout the schema.
• Reference: It serves as the single source of truth for attribute definitions, so any event class or object in OCSF refers back to the dictionary for its attribute specifications.
• Validation: Tools and schema validators use the dictionary to check that event objects conform to expected types, required/optional status, and allowed value sets.
• Documentation: Each entry includes a caption and a description, which helps both humans (developers, analysts) and machines understand the meaning and usage of each attribute.

For example, the dictionary defines what actor, activity_id, cmd_line, and hundreds of other attributes mean, their types, and any enumerated values or special rules. This centralized approach is crucial for interoperability and extensibility across different cybersecurity products and data sources using OCSF.

Problems

• Since we have 3 class families instead of 1 and especially with the presence of features there are terms from multiple areas put together in a single dictionary, where terms with multiple meanings, or different meanings in different contexts would be hard to avoid in the long run.
  • e.g. signature in the agent record object cannot mean the signature object of object_t type, and the actual bytestring_t type singature itself in the signature object
  • or data cannot be a specific object of object_t and just an unspecified JSON of json_t type at the same time.
  • data or dependencies that refer to different objects depending on where it is used can only be added currently with a workaround (overwriting hidden values from the dictionary at the object level) which is not intuitive
• Contributors have a harder time adding every new attribute also to the dictionary, where different attributes are required than in the objects (requirement goes to the object, type goes to the dictionary)
• Not clear who and why would someone browse the dictionary in the case of OASF

Consequences for removing the dictionary

• Type and all the other info about the attribute would be specified in the object’s json descriptor file, which would make it easier to contribute to the schema (no need to create a new attribute at two places, all of the information about an attribute would be at the same place).
• Generic attributes like data, dependencies etc. that mean different objects at different places could be called without any prefixes or workarounds.
• The same objects would still be referred in the attributes, only the naming of said attributes would become flexible
• It would be more computation-heavy to link the objects to each other building up the graphs maybe?

[akijakya]

The problems could also be overcome by using a slightly more flexible system, without removing the dictionary:

1. Make attribute types overwritable by the type in the class/object definition.
   Difficulty: Low
   This is not optimal in my opinion, as it breaks consistency between the dictionary and the objects.

2. Keep the dictionary for attributes meant to be reused, but add the ability to define attributes in the class/object definition files without adding them to the dictionary.
   In this case, a marker or filter could also be added to the attribute on the class/object page to indicate whether it is unique or reusable.
   Difficulty: Low
   This is not optimal either, as it leaves the responsibility to add or reuse attributes from the dictionary to contributors, and still leaves the overhead without enforcement to update the dictionary. Gradual abandonment of the dictionary can happen.

3. Don’t use a dedicated dictionary file in the schema; put everything in the class/object definitions, but collect all attributes at runtime and display them on a page, merged together when the name and type are the same.
   The menu title could be called “Attributes” instead of “Dictionary.”
   Difficulty: High

4. The same as option 3, but there is a dictionary file which needs to be updated with every new attribute. The class/object-specific versions of an attribute are added separately using dot-notation, like signature (generic) and agent.signature (specific version for the agent object).
   Difficulty: Medium

5. Separate OASF reference names from the names being displayed in the class/object JSON.
   So, although it appears in the dictionary as agent_signature, it will be rendered as signature in the JSON. A new field needs to be added to attributes where the reference name is different from the actual name; otherwise, the reference could default to the attribute name.
   Difficulty: Low

The more I think about it, the more I like option 5, as the implementation only needs an additional check for the reference field in an attribute to connect it with the dictionary entry and its expected type.

[ramizpolic]

Very detailed summary on the issues, thanks @akijakya. I believe the 5th item really addresses this problem nicely because a) it extends the schema in a non-breaking way, and b) addresses the limitation of unique property vs type names. With that, we pretty much can have arbitrary property names without being limited to the actual name of the type. I like it! I'd suggest we move with that implementation.

[akijakya]

I'm glad you are also happy with option 5, issue created for the implementation: #150