KiKit CLI

[yaqwsx]

Hello guys, I have a question for you.

Currently, I can think about many new features KiKit could do. However, I struggle with designing a suitable CLI for them. Currently, the panelize CLI is really clumsy and hard to get right on the first shot. There are so many options and I think more of them will come in the future.

So I would like to ask you for a help; to brainstorm. Consider the current KiKit CLI for panelization; what would you change? How would you redesign it to make it easier to use?

[jdaandersj]

I would consider an approach with make file. This way you can keep pure CLI for super simple things, make file for advanced stuff, and still the python directly if you want to have full control. Added bonus that the make file is easy to track in git.

Also, thanks for the really useful tool!

[yaqwsx]

Thanks, what do you mean by "with makefile"? Could give an example? Currently, I use Kikit in Makefiles - but that invokes CLI (e.g., https://github.com/RoboticsBrno/RB0004-NeopixelBooster/blob/master/Makefile).

If we specify a custom file format, I think it will face similar issues as command-line options - there will be too many options. It will be roughly equivalent to writing a shell script with the CLI option (with more or less convenient syntax). But it will still be the same.

[jdaandersj]

Ah I was thinking along similar lines to dockerfiles.

Yes I guess this is very similar to just having a script but the added structure would help a lot with readability. Edit: and also is probably easier to document

[danielkucera]

I would suggest making it a plugin for KiCad. This would of course mean the UI has to be designed but it would be much easier to use for users which are scared by CLI

• you wouldn't have to design CLI
• you could just leave it as python module for usage from CLI by advanced users

[FFY00]

CLI would still be desirable.

[yaqwsx]

I also don't want to give up CLI. It is very convenient to be able to write it directly in makefile without extra file. I still would like to be able to create the most common panel types from CLI.

The work on GUI was already started; panelization is, however, not implemented yet. Also, I still perceive KiKit as an automation tool, so in my opinion, GUI is nice to have, but it is not the primary target of KiKit. In my opinion, the "standard user of KiKit" wants to create panels in CI and makefiles. I don't want to force myself or others to write (even short) Python scripts for such repetitive tasks.

[yaqwsx]

I have a proposal for the new KiKit UI; I am happy to hear your thoughts.

The problem

Current KiKit UI is messy - everything is an option, some of the options take multiple numbers and it is really hard to guess what the number means. The naming of the options is often quirky in order to make them unique and thus, the options are often unintuitive. Also, it is not clear what parameter of the panel the options refer to. New options contribute to this mess.

It is also not easily possible to reuse panel settings between projects as everything is captured in the command line arguments.

The solution

Panel options can be easily grouped; e.g., we have options specifying layout, tabs, framing and cuts. We could specify all these options into a JSON file:

{
  "layout": {
    "style": grid,
    "cols": 3,
    "rows": 4,
    "spacing": "4mm",
    "alternation": "none"
  },
  "source": { // Specify the source area
    "style": "auto", // Could be also rect; e.g., polygon in the future
  },
  "tabs": {
    "style": "normal", // Also, could be e.g., full width
    "source": "auto", // Also, could be layer or annotation
    "width": 3, // If no unit is specified, assume millimeters
    "hcount": 1,
    "vcount": 1
  },
  "framing": {
    "style": "railsTb", // none, full, railsLr, frame
    "railWidth": "5mm"
  },
  "tooling": {
    "style": "3hole", // none, 4 hole
  },
  "cuts": {
    "style": "vcut", // mousebites
    "layer": "User.Cmt"
  }
}

KiKit could be then invoked via kikit panelize --preset myPreset.json --layout 'rows: 5, cols: 6' --cuts 'mousebites, hole: 0.5mm, spacing: 0.8mm, offset: 0.25mm'.

Basically, the preset is the source for the settings and then you can locally override the parameters via command-line options as key-value pairs. The options are named according to the groups. Note that every group has a style parameter that dictates the style of the feature.

KiKit could then ship with default presets; e.g., panelize with V-cuts, or panelize for JLC PCB SMT assembly. We could mark such predefined presets with leading :; i.e., kikit panelize --preset :jlcAssembly. That means that the user needs to only override a few of them to get a decent result. Also, the users can create custom presets and easily share them across projects; e.g., via GitHub (would it be interesting to be able to specify preset as URL for the file?). There should be an implicit, safe, preset (when the user does not specify any).

The preset JSON should not be nested - it would make overriding the options complicated. If nesting is desired, we could emulate it with key values separated by .; e.g., "option.nestedOption". Whenever possible, we should avoid this or maybe prohibit it.

The syntax for overriding the options is the following: it is a list of key-value pairs separated by a ,. The key is separated from the value by :. All the options are specified as strings (therefore, we do not enclose them in quotes). The users are encouraged by documentation to specify the option in single quotes to prevent the shell from escaping and interpreting whitespace. The "style" key might be ommitted for brevity (but also, have to be the first word) - that instead of --cuts 'style: mousebites, ...' your could write just --cuts 'mousebites, ....'.

Why is this a good idea?

• the options are grouped
• the commands might get shorter due to the presets
• we can easily specify units, no more cryptic tuples of numbers
• there will be "verified" panelization options for fabhauses; just like with fab command
• it is easy to add new options and styles without breaking the CLI
• it opens possibilities for adding new layouts without being clumsy (we could introduce KiKit plugins for community maintained layouts)
• the users can share their presets in a convenient form (JSON file) (e.g., on the planned showcase page).
• it is easier to share common parts of the code (currently, there is a non-trivial duplication between grid and tightgrid styles).

Why is this a bad idea?

• this is a breaking change (on the other hand, it is the possibility to finally release KiKit 1.0)
• overriding is a little bit more verbose than it used to be
• we need proper documentation for the options; it would be no longer possible to cover them in built-in help (however, the question is if the current form of help is useful or not - so maybe there will no change in the usefulness of the built-in help).
• implementing shell completion might be tricky (but should be doable with Click)

Possible options

• kikit panelize extractboard could be deprecated in the favour of kikit panelize --preset :extract; or it could become an alias
• we could allow the user to specify multiple presets; the specified values override options from the previous presets.
• we could have an option --bake file.json which would take a preset, overrides and baked the overrides in the file.json - for easy sharing of the presets.

Please, express your opinions!

[yaqwsx]

Thank you guys for your responses; I appreciate them a lot.

• the examples page will probably not change - it will only adapt to the new CLI. We will probably document the built-in configurations on a separate documentation page.
• in general; I am not a huge fan of YAML because of its loose syntax and ambiguity (in my experience you can easily burn yourself). This is why I proposed JSON. There are already JSON extensions that allow for comments (basically, you can annotate JSON by C-like comments and strip them before you put it in the parser). This would be preferred way if the only thing against JSON is the absence of comments. JSON is more verbose compared to the YAML (because of brackets and quotes) but has nice strict syntactic rules.
• I like the -p option.
• I do not plan to support web app. That makes no sense in my opinion. Instead, I plan an Action plugin for KiCAD. Basically, you would open an instance of PcbNew with no board. You will press a button that will invoke a form where you can basically specify the JSON configuration via GUI. You can click on a button and generate the current configuration (thus, you get a "preview" in PcbNew). You can also save the presets as JSON (e.g., to use them in a makefile). Since the generation of the panel usually takes about 1 second, it will not be an instant preview, but close enough.

[jdaandersj]

I also dislike YAML so if JSON comments can be done I am in favour of that approach!

[JulienDevillers]

I Agree.

[sslupsky]

Basically, you would open an instance of PcbNew with no board. You will press a button that will invoke a form where you can basically specify the JSON configuration via GUI. You can click on a button and generate the current configuration (thus, you get a "preview" in PcbNew).

I suggest starting from the board view. That is, pcbnew is open with the board you want to panelize. You press a button that will invoke a form to set/adjust the settings via GUI. You click a button to generate and the plugin generates the panel and opens a new instance of pcbnew with the panelized board.

[sslupsky]

-p option is a good idea.

Speaking of cuts.style, it would be nice if you could control the cut style on a per tab basis. So, some tabs could be vcuts and others could be mouse bites. Maybe the cut style could be a parameter of the tabsfrom option?

[aa956]

Proposal to add placements of arbitrary footprints to arbitrary locations, something along the lines of:

{
    "tabs": {
        "type": "normal",
        "width": "3mm"
    },
    "framing": {
        "type": "frame",
        "frameWidth": "100mm",
        "frameHeight": "100mm"
    },
    "footprints": [
        {
            "type": "./panel.pretty/fiducial_round_1mm_2.5mm.kicad_mod",
            "x": "5mm",
            "y": "5mm",
            "layer": "F.Cu",
            "angle": "0°"
        },
        {
            "type": "./panel.pretty/fiducial_diamond_1mm_3.0mm.kicad_mod",
            "x": "95mm",
            "y": "5mm",
            "layer": "F.Cu",
            "angle": "0°"
        },
        {
            "type": "./panel.pretty/fiducial_square_1mm_3.0mm.kicad_mod",
            "x": "95mm",
            "y": "5mm",
            "layer": "B.Cu",
            "angle": "0°"
        },
        {
            "type": "./panel.pretty/pcb_part_number.kicad_mod",
            "x": "45mm",
            "y": "7mm",
            "layer": "F.Cu",
            "angle": "0°"
        },
        {
            "type": "./panel.pretty/pcb_barcode.kicad_mod",
            "x": "50mm",
            "y": "95mm",
            "layer": "B.Cu",
            "angle": "0°"
        }
    ]
}

It may help with these:

[ ] Implement tooling (#91, #123)
[ ] Implement fiducials (#127)
[ ] Implement text labels (#109)

Exact syntax may need some thought.
It may be better to use "side": "front" instead of the "layer": "F.Cu".

[yaqwsx]

Thank you for the proposal. I am not sure about this one. It feels like it brings a lot of complexity to the UI (and also requires nested configuration - which I want to really avoid). Although I agree that adding e.g., a barcode is useful, but supplying tooling holes implementation via this seems rather hacky to me.

What I see as a problem, are the many ways we can specify the position of the module - relative to the four corners, relative to the rail, frame, etc. That brought me to another idea - what if the user could supply a piece of Python script that will be run after the main panelization procedure finishes? KiKit API is rich, you have the full Python power, so you can express anything you need. There are already KiKit's users who write their custom panelization scripts, however, they would often benefit from not having to construct the initial panel.

About tooling holes implementation; once we have #139 implemented, it is pretty easy to implement new styles, e.g., of tooling holes.

What do you think?

[yaqwsx]

#139 was merged to master. Please test it and report any problems you find.

[rokan51]

Or minimalist approach: bundle everything in a single parameter and use the same parser for config file and command line args:
kikit COMMAND [--options] "Your json-without-outer-braces string" INPUT OUTPUT
like
kikit panelize "layout: {style: grid, cols: 3,... }, tabs: {width: 3,... }" input.kicad_pcb output.kicad_pcb

If kikit would be invoked with only four parameters, "--options" (-o) : might be assumed to be omitted making command shorter. This would be also compatible with your current implementation if you want to keep it, for "overspecification" simple priority could take place for implementation simplicity of merging json-s.

[yaqwsx]

I assume that the option -o makes sense if you have already stored your configuration into a JSON file. Otherwise, it is in my opinion much less pleasant to write a JSON as a command-line argument as there are more special chars (and shell can eat them which is pain to debug). So if you have the file, why not just supply it via -p? Or do I miss something? What is so appealing in writing -o '{"layout": {"x": "y", "z": "a"}}' instead of simple--layout 'x: y; z; a'?

[rokan51]

Yeah, I just thought that your syntax somewhat resembles json (although you use single quotes instead of {} brackets and semicolon instead of comma), it might be possible to unify the syntax of config file and command-line and keep only one parser. I forgot that quotes are REQUIRED by json which makes things messy with those pesky quotes and their treatment by the command-line shells when passed around.
Although it would be possible to accept also dirty-unquoted json (eg. most primitive way assuming no quotes in keys/values be to quoting it first with something like re.sub("(\w+)", r'"\1"', dirty_arg) before feeding it to json lib), however your way seem to be better.
Anyway I've just discovered kikit recently, congrats for an excellent tool.

[yaqwsx]

The quotes in the examples are because of the shell - otherwise, the shell splits the parameter string into multiple arguments. Concatenating the arguments back into a single string is not (at least easily) possible with the wonderful click library we use for CLI. Also, it might be ambiguous how to concatenate the strings back. So I decided to avoid it. Thus I introduced a simple syntax for CLI specification (which allows you to commit e.g., the style keyword).

Quoting JSON back is again, ambiguous, so I would stay away from it. I usually prefer to be explicit over having a magick that can easily break in surprising ways (especially on mistyped characters).

Happy to hear you like KiKit.