[CI] Make linting a separate job

[asfimport]

When pushing changes I get routinely frustrated because Travis-CI fails early on a linting failure. It would be nice if linting (Python, C++) was a separate job so as not to disrupt the development workflow so much.

Reporter: Antoine Pitrou / @pitrou
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #2831

_{Note: This issue was originally created as ARROW-3260. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
I've create a PR for reproducing CI build via docker compose: #2572

So something like the following is possible:

docker-compose run python flake8

We could define the linux CI builds in dockerfiles and just simply run them on travis. We could even benefit from docker's caching mechanism.

[asfimport]

Antoine Pitrou / @pitrou:
That sounds orthogonal to what I'm proposing.

[asfimport]

Krisztian Szucs / @kszucs:
It's indeed orthogonal, however in order to not increase the CI runtimes We should run all of the linters in a single CI step.

[asfimport]

Wes McKinney / @wesm:
Seems like having a more accessible pre-commit hook could help. I run the following code before every push to github

function arrow_preflight {
    ARROW_PREFLIGHT_DIR=$ARROW_ROOT/cpp/preflight-build
    mkdir -p $ARROW_PREFLIGHT_DIR
    pushd $ARROW_ROOT/cpp
    python build-support/lint_cpp_cli.py src || return
    popd
    pushd $ARROW_PREFLIGHT_DIR
    arrow_cmake
    ninja format
    ninja lint || return
    popd
    pushd $ARROW_ROOT/python
    flake8 --count pyarrow || return
    flake8 --count --config=.flake8.cython pyarrow || return
    popd
}

[asfimport]

Antoine Pitrou / @pitrou:
I don't think, I think it would be nice to be able to treat linting as a last polish step when a PR is ready, not as something you must do each time you push a new change to the CI (you may automate linting in a commit hook but you still have to fix lint errors manually).

[asfimport]

Wes McKinney / @wesm:
I'd be fine with moving RAT checks and linting to a separate build step.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 2831
#2831