Skip to content

Add Makefile to generate python files based on idl definition #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

timl3136
Copy link
Member

@timl3136 timl3136 commented Jul 24, 2025

Summary

Adds a Makefile that automates generation of Python protobuf files from Cadence's IDL definitions. Processes both API and Admin protobuf files from idls/proto/uber/cadence/ and generates corresponding Python _pb2.py files in .gen/ directory.

Changes

  • Makefile - Build automation with targets: all (default), clean, help, show
  • 20 generated Python files in .gen/uber/cadence/api/v1/ and .gen/uber/cadence/admin/v1/

Usage

make        # Generate all Python proto files
make clean  # Remove generated files
make help   # Show available targets

Prerequisites

  • protoc (Protocol Buffers compiler) in PATH
  • Source .proto files in idls/proto/uber/cadence/

Makefile Outdated
# Generate Python files for API protos
$(DST_DIR)/uber/cadence/api/v1/%_pb2.py: $(PROTO_ROOT)/api/v1/%.proto | $(DST_DIR)/uber/cadence/api/v1/
@echo "Generating Python file for $(PROTO_ROOT)/api/v1/$*.proto"
protoc --proto_path=$(SRC_DIR) --python_out=$(DST_DIR) $(PROTO_ROOT)/api/v1/$*.proto
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably want to copy the protoc download-and-init-in-temp-build-dir thing from the other cadence repos :\ protoc's code is far from stable across versions.

Copy link
Member Author

@timl3136 timl3136 Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

protoc v3.14.0 does not have pyi support, should I just a separate version for the pyi files? I think .pyi files are quite essential for dev to understand type definition.

Copy link
Member

@Groxx Groxx Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no need for it to be the same version as go/java, we can just pick a newer one. they're all wire-binary-compatible with each other, and we have no code-API backwards compatibility needs that would force us on an older version (unlike go/java)

Makefile Outdated
show:
@echo "Proto files to process:"
@echo "API:"
@for file in $(API_PROTO_FILES); do echo " $$file"; done
Copy link
Member

@Groxx Groxx Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

possible alternative to these, for makefile-debugging purposes:

$(info $(API_PROTO_FILES))

^ basically anywhere, but generally at parse-time (outside any target).

(tbh I think these "help understand the makefile" targets are completely fine while things are simple, so this LGTM as-is. just offering an alternative for future debug-sessions. remake is also truly excellent and highly recommended)

@Groxx
Copy link
Member

Groxx commented Jul 25, 2025

copying from side-chat:

  • yea seems reasonable, happy to approve if it'll streamline anything.
  • versioned-protoc seems probably critical eventually. it's a really big pain as time passes.
  • .pyi files seem like a good idea to include
  • it's currently written with every file doing a separate protoc run. in other languages that tends to cause problems if you make protoc -j to run them in parallel, because it has a nasty habit of clobbering other files / reading partially-written files and getting confused.
    • is that an issue in python? if no, the setup now is ideal. if yes, we should probably pull it into a single protoc-target that runs them all in a specific way.

make -j seems fine, so 👍 that's a nice way to do it.

also we might want to pull in the submodule-check from the server repo, since this silently does nothing if the submodule isn't initialized.
(also entirely fine as-is for now, just future polish stuff)

@timl3136 timl3136 requested a review from Groxx July 25, 2025 20:30
@shijiesheng
Copy link
Member

Python uses pyproject. Please refer to https://github.com/temporalio/sdk-python/blob/main/pyproject.toml on how to add build-system.

@timl3136 timl3136 closed this Aug 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants