-
Notifications
You must be signed in to change notification settings - Fork 194
Design document for Node Interface Definition Language (IDL) #266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: gh-pages
Are you sure you want to change the base?
Changes from 15 commits
dccd234
e8e0247
b8cc64a
3c8028c
f8ebe30
ee4ee96
80ff88b
fce235c
45d4919
99a3e7c
ed7aedb
78f4d27
23aeab0
3fb7ab5
c97a7a9
da63aa6
19b9738
2da495a
ef5018b
3642623
83a4715
2a50c44
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| ```xml | ||
| <interface version="1"> | ||
| <node name="node_1" > | ||
| <parameter name="verbose" type="bool" /> | ||
| <topic name="chatter" type="std_msgs/msg/String" publisher="true" /> | ||
| </node> | ||
|
|
||
| <node name="node_2" > | ||
| <parameter name="rate" type="int" /> | ||
| <topic name="/foo/bar" type="std_msgs/msg/String" subscription="true" /> | ||
| <service name="/example_service" type="std_srvs/srv/Empty" client="true" /> | ||
| <action name="/example_action", type="example_interfaces/action/Fibonacci" client="true" /> | ||
| </node> | ||
| </interface> | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| ```xml | ||
| <package format="2"> | ||
| <name>example-package</name> | ||
|
|
||
| <!-- snip --> | ||
|
|
||
| <export> | ||
| <interface path=”${prefix}/interface_declaration.xml”/> | ||
| </export> | ||
| ``` |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,205 @@ | ||||||||||||||||||||||||||||||||||||||
| --- | ||||||||||||||||||||||||||||||||||||||
| layout: default | ||||||||||||||||||||||||||||||||||||||
| title: ROS 2 Node Interface Definition Language | ||||||||||||||||||||||||||||||||||||||
| permalink: articles/ros2_node_interface_definition_language.html | ||||||||||||||||||||||||||||||||||||||
| abstract: | ||||||||||||||||||||||||||||||||||||||
| This article specifies the ROS 2 Interface Definition Language, a simple and standardized manner to export the complete interface (action/message/parameter/service) of node(s) in a package. | ||||||||||||||||||||||||||||||||||||||
| author: > | ||||||||||||||||||||||||||||||||||||||
| [Jérémie Deray](https://github.com/artivis), | ||||||||||||||||||||||||||||||||||||||
| [Kyle Fazzari](https://github.com/kyrofa) | ||||||||||||||||||||||||||||||||||||||
| published: true | ||||||||||||||||||||||||||||||||||||||
| categories: Interfaces | ||||||||||||||||||||||||||||||||||||||
| --- | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| {:toc} | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| # {{ page.title }} | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| <div class="abstract" markdown="1"> | ||||||||||||||||||||||||||||||||||||||
| {{ page.abstract }} | ||||||||||||||||||||||||||||||||||||||
| </div> | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Original Author: {{ page.author }} | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ## Background | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Every ROS 2 node has an associated interface that describes how it communicates with other nodes, as well as how it is to be configured. | ||||||||||||||||||||||||||||||||||||||
| This interface is defined in code, and consists of: | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| - Actions (server or client) | ||||||||||||||||||||||||||||||||||||||
| - Parameters | ||||||||||||||||||||||||||||||||||||||
| - Services (server or client) | ||||||||||||||||||||||||||||||||||||||
| - Topics (publisher or subscriber) | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| The information contained within that interface is obviously very valuable on different levels and from different perspectives. | ||||||||||||||||||||||||||||||||||||||
| While it is usually readily available to a developer looking at the code, it cannot reliably be automatically extracted. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
| It therefore calls for the creation of a standardized way to explicitly define and export this information. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| This article defines a high-level abstraction allowing upstream packages to specify the communication requirements of the nodes in the package, such that the final user, be it a developer or a static analysis tool, can benefit from it. | ||||||||||||||||||||||||||||||||||||||
| The Interface Definition Language (IDL) specified in the next section is meant to be distributed alongside its associated package, be it in the source code or a generated release packaging format (e.g. debian). | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
| Whether the interface is declared or not is up to the package author and should not prevent the correct execution of any system pre-existing the IDL. | ||||||||||||||||||||||||||||||||||||||
| Similarly, the declared interface may be only partial and allow for the full use of pre-existing systems and the use of dependent systems on the parts covered by the partial interface. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ## Motivation | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| While initially being approached from a ROS 2 Security perspective, the abstraction level of the IDL allows for the developments of other functionalities and tools. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ### Security motivation | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Thanks to [DDS-Security][dds_security] and [SROS 2][sros2_design], security is at the heart of ROS 2. | ||||||||||||||||||||||||||||||||||||||
| DDS enforces access control using a signed permissions document for each domain participant (see the [DDS-Security spec][dds_security], section 9.4). | ||||||||||||||||||||||||||||||||||||||
| In SROS 2, that permissions document is generated from a ROS-specific [XML][xml_wiki] policy file that may include the permissions for one or many nodes. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Currently policy files can be created in one of two ways: | ||||||||||||||||||||||||||||||||||||||
| - Written by hand. | ||||||||||||||||||||||||||||||||||||||
| - A snapshot of the live ROS 2 graph can be taken and written into a policy that covers its current state via `ros2 security generate_policy`. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| While the first option is obviously very tedious and error-prone, the second only partially alleviates the burden due to the fact that it cannot fully cover the dynamic nature of a ROS 2 graph and all of its interactions. | ||||||||||||||||||||||||||||||||||||||
| More problematic than these issues, though, is that both options put the onus of security squarely on the shoulders of end users. | ||||||||||||||||||||||||||||||||||||||
| This introduces two problems: | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| While developers will be able to define the set of rules securing their own ROS 2 nodes, the nodes developed in-house are often outnumbered by upstream components when it comes to the entire node graph, and the developers are typically not experts in every component being used. | ||||||||||||||||||||||||||||||||||||||
| Without that expertise, the entire node graph cannot be properly locked down. | ||||||||||||||||||||||||||||||||||||||
| Consider a complex and popular upstream component, perhaps parts of the navigation stack. | ||||||||||||||||||||||||||||||||||||||
| **Every end user** of this component must duplicate the effort of attempting to properly lock it down. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| If ROS 2 provided a way for upstream package authors to specify the interface required by the nodes in their package, and if the tools to generate security policies from that interface existed, neither of these problems would exist. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| ### Other motivations | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| Outside of security, there are several fascinating possibilities unlocked by having such an interface. | ||||||||||||||||||||||||||||||||||||||
| For example, consider how this could impact [ROS 2 launch][launch_ros]. | ||||||||||||||||||||||||||||||||||||||
| It would be able to statically (i.e. before running anything) determine if parameter names or remappings are incorrect, among other similar sanity checks. | ||||||||||||||||||||||||||||||||||||||
| Another example of the usefulness of having a static interface is the ability to create graphical tools for putting a ROS system together. | ||||||||||||||||||||||||||||||||||||||
| Yet another example would be an additional feature in `ros2 pkg create` that would allow a developer to hand it an IDL and have it generate scaffolding for a node with that interface. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
| These examples are only a subset of use-cases made possible by such an interface. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
| It's clear that this is useful well beyond security. | ||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
| ## Challenges to overcome | |
| This proposal has a number of potential upsides, but it also has some downsides worthy of discussion. | |
| ### This is only really useful if it gains significant adoption in upstream packages | |
| This is a valid criticism, particularly from a security standpoint. | |
| However, its usefulness hopefully outweighs the work required to implement it upstream, and it's certainly something that can be contributed by community members given that the interface would be reviewed by the experts in the package. | |
| ### Declared and actual interface can get out of sync | |
| This is certainly a concern: an out-of-date interface is debatably less useful than having no interface at all. | |
| One potential solution for this is to more tightly couple the declared and actual interface by creating a library that consumes the declared interface and creates the corresponding ROS entities. | |
| That has its own challenges, of course (e.g. how does it deal with wildcards?), and would need to be properly designed. | |
| This is an area of future investigation beyond the scope of this article. | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if those concerns belong to a design doc...
We therefore keep them as suggestion here until we figure that out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's worth adding potential issues and how we plan/could address them.
I'm not convinced that adoption is a challenge. I'd argue a node IDL is still useful to individuals, or organizations using it internally.
Regarding sync issues, we could at the very least have a tool that launches each node and does a best-effort sanity check against the IDL (maybe this is what you are describing, I couldn't tell). It could report any discrepancies between the declared and actual interfaces. I bet this check could be added as a unit test in a lot of cases. Beyond that, I think we just have to rely on authors and community reports of inconsistencies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. I'll keep this as suggestion until updated with plan to address them.
The adoption challenge mentioned here is maybe a little too security (future plan/tools) centric and its tone could be voiced down a little.
As for your suggestion on the sync issue it will be added as potential plan to address it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have a verb under ros2 nodl such as verify. Make it really quick and people could potentially run it regularly at runtime, e.g. once a minute, as a sanity check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright, updated suggestion with what was discussed here and pushed it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I discussed this with some people, but I don't recall who was there, perhaps @ruffsl and @thomas-moulard. We could create an extension to the Node concept (much like LifecycleNode) that consumes the NoDL file as part of the constructor (or early in execution) and would not allow topics/services/parameters/actions outside of that policy to be created either at all or past a certain point in execution (perhaps coupled with the configure state of LifeCycleNode.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would be an excellent way to ensure these stayed in sync. It's essentially what I was thinking when I mentioned that this could live within RCL (the next sentence, I think). The only thing I dislike about the idea is that it still requires the developers to duplicate the information: once in the NoDL and again in code. It would really be ideal if they only had to write it once, but that's a harder problem to solve.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think to solve that you would have to take the same approach as messages, and resort to generating base classes for node implementations to inherit from. At this point you are starting to get well into the realm of model-based tool chains and all the benefits and risks that entails. One big risk is the restrictions it places on developers in how they write and manage their code, which can become a barrier to adoption due to the increased learning required. Tools that have done this in the past have seen limited adoption while ROS has been happily adopted by people all over the place.
It's difficult to say which is the correct way to go because there are many facets to the decision.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does it have to be tied to the package? Or the node, for that matter? I think it could be useful to distribute node interface descriptions separately and have several different nodes from different packages that support the same interface.
In computer science, this is one of the primary reasons for using interfaces: it allows substituteability. Particularly if you are going after static analysis tools and system constrution tools, such capability would be close to essential.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a nice idea. Then there's the question: how to communicate that your node implements a particular NoDL?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could have common packages that just contain NoDL files, e.g. sensor_nodls, nav_nodls, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah how one would tied node and NoDL back together if they are distributed separately. Also I believe that people are somewhat familiar with the plugin export scheme, so a look alike scheme would ease adoption.
Finally separating NoDL files in a separate pkg(s) just increase the chances that the files go out-of-sync.
I do see and appreciate @gbiggs' point on having some kind of template NoDL files tho.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@artivis I think you are looking at this too much from the point of view of describing existing nodes, and not enough from the point of view of describing types. Describing types is ultimately what you are doing, and I think the correct approach is to find a way to state that a node implements a particular NoDL interface rather than saying that every node is a completely unique, unsubstitutable type that needs its own NoDL file.
In some ways this is a philosophical question about where the substitution level is in ROS. There are probably those who say it is at the level of topic, and those who say it is at the level of node. There are definite benefits to doing the later, but it does require more careful engineering and/or tools to support and ROS users may not have an appetite for that. I would argue, though, that if they are at the level of caring about proper security they are probably at the level of caring about good engineering practices and so wouldn't mind some extra complexity if it gives useful benefits, like powerful static analysis tools.
However I think the most important is that the reality is that people treat many nodes in this way already, and the state publisher node discussion on Discourse is a currently-being-discussed example of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this context, I think that describing nodes versus types is a distinction without a difference. In the end, node A has interface X, regardless of where X comes from. As @ruffsl points out in another thread, we can xinclude files from other packages if we want to declare X in one place and have multiple packages/nodes use it. I don't see a solid use-case for designing much more than that at this time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not asking for designing too much, but I think that we need to make sure what we design now does not close off future potential. Having a way to define node interfaces, parameters a node can take, etc. outside of the source code opens up huge possibilities for better engineering practices for ROS users, but it needs careful thought for what exactly is defined and how to ensure that we don't make it impossible to do, for example, easy substitution of equivalent nodes, or nodes with optional extensions to a defined interface, or verifying the compatibility of two nodes at launch time or in some pre-launch system check.
jacobperron marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
jacobperron marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might also be worth mentioning that there could be one or more instances of this tag.
I could imagine a package author wanting to describe their nodes in separate files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you elaborate on this? I tend to lean toward the simplest system that will work instead of immediately trying to enable everything a developer would possibly want to do. Having multiple interface files is a complexity that I'm not sure is currently warranted, particularly given the ability to xinclude. We can of course add it, I just want to learn more about what's in your head here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking that we could define nodes interfaces in one or more files. This comment thread is more relevant: #266 (comment).
In that case, we might expect there to be one or more occurrences of this tag in the package.xml.
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was not aware that ROS 2 had such a tight coupling between packages and nodes.
dirk-thomas marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why should multiple independent nodes be described in the same file? I would think a one node-per-file mapping would be much more natural since they can evolve independently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that it's more natural to have one node per file. I think both option should be possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In our mind this mechanism is very similar to that of plugins and therefore both options are possible. I'll make this explicit in the text.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that allowing multiple nodes in a file is something you would come to regret.
I also think that if you are allowing mutliple nodes in a file, then you are describing a Package IDL, not a Node IDL. Although this comment applies generally to the tone throughout this design document.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gbiggs Do you have an example to illustrate this? As long as one can parse the info on a per-node basis I don't see any problem.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With XML, one could always xinclude from external paths, such that you could break your "package" IDL into separate node "IDL" files, if one wanted to.
jacobperron marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ROS 2 introduced read only parameters. Should the parameters attributes includes 'ro/rw' ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO, it's okay to not include the read only status. We can already query this constraint on parameters (and other constraints) at runtime. Unless there's a strong case for static analysis, I feel like describing it in the node IDL just adds another point where information can go out-of-sync.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be good if the parameter description could be separated. I could see it being read independently by a node which doesn't have a "Node IDL" but wants to read the declared parameters from a file. That would imply that the parameter would need several additional arguments though (all which are possible to pass using the parameter declaration API: ranges, read-only, ...).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jacobperron Not that one of the point of this proposal is to access the API info in a 'pre-runtime' fashion - e.g. at launch-parsing time. So querying for ro/rw and other constraints at runtime is kinda too late already.
@dirk-thomas I agree that it might be interesting, but it does feel a little out of scope here. Do you mean something a la ROS 1 rosparam_handler?
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since the QoS settings are deciding of two endpoints on the same topic name are being matched wouldn't it make sense to also specify those? Otherwise any "connectivity analysis" on these metadata might be incorrect.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair point, going to update that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added the QoS settings for actions, services and topics (da63aa6). I'm not familiar with the later two so please let me know your take on this.
I also went ahead assuming we would like those to defaults to the same values they do in code, hopefully I'm correctly linking to the defaults.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Forgot to mention, I don't know what type should the qos attribute be. I'm not sure of to translate it to a xml tag attribute here...
If anyone has a suggestion?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that they are objects in code, it may not be suited to being an attribute at all, but an object nested under the respective tags.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean something similar to <rosparam> in roslaunch?
E.g.
<topic name="/foo/bar" type="std_msgs/msg/String" subscription="true">
<qos>
policy_history: best_effort
policy_reliabiliity: reliable
...
</qos>
</topic>There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's the right idea, yeah, although I'd make those QoS attributes, maybe something like:
<topic name="/foo/bar" type="std_msgs/msg/String" subscription="true">
<qos policy_history="keep_last" policy_reliabiliity="reliable" \>
</topic>
I also went ahead assuming we would like those to defaults to the same values they do in code
I think this makes sense. Given that the rationale for the defaults is "keep behavior like ROS 1" I doubt they will change and get out of sync.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated qos so that it is a nested object in action, service and topic.
Updated the example to highlight qos. The example is not fully finished.
Uh oh!
There was an error while loading. Please reload this page.