As a part of upgrading Kedro from 0.16.2 to 0.17.3 in our organization, I've made changes to Kedro related files in our codebase based on Kedro starter pyspark-iris on 0.17.3.
Now I get an error of Error: No such command 'run' on kedro run.
setup.py
from setuptools import find_packages, setup
entry_point = "kedro-project = kedro-package.__main__:main"
# get the dependencies and installs
with open("requirements.txt", "r", encoding="utf-8") as f:
# Make sure we strip all comments and options (e.g "--extra-index-url")
# that arise from a modified pip.conf file that configure global options
# when running kedro build-reqs
requires = []
for line in f:
req = line.split("#", 1)[0].strip()
if req and not req.startswith("--"):
requires.append(req)
setup(
name="kedro-package",
version="0.1",
packages=find_packages(exclude=["tests"]),
entry_points={"console_scripts": [entry_point]},
install_requires=requires,
extras_require={
"docs": [
"sphinx~=3.4.3",
"sphinx_rtd_theme==0.5.1",
"nbsphinx==0.8.1",
"nbstripout==0.3.3",
"recommonmark==0.7.1",
"sphinx-autodoc-typehints==1.11.1",
"sphinx_copybutton==0.3.1",
"jupyter_client>=5.1.0, <6.0",
"tornado>=4.2, <6.0",
"ipykernel~=5.3",
]
},
)
main.py
from pathlib import Path
from kedro.framework.project import configure_project
import logging
from .cli import run
def main():
package_name = str(Path(__file__).resolve().parent.name)
logging.getLogger(__name__).info(f"package name is: {package_name}")
configure_project(package_name=package_name)
run()
if __name__ == "__main__":
main()
and cli.py is at the same level as main.py which are directly inside the package (altered to kedro-package here for anonymity)
This only happens when performing kedro run on the EMR. When we run locally we don't see that error. Rather it errors out because it can't connect to S3, which is expected. Additionally, I've tried running