8:30 a.m.
(Long mail, question at end.)
I set up a demo instance of git and invited Daniel to try it. (If anyone
else wants to have a go, let me know privately.)
One of the issues Daniel found is that ick does not yet support well is
building a source tree from a stack of git repositories. This is something
Daniel does to keep his upstream code and its Debian packaging separate.
Basically, the upstream code lives at:
git://git.example.com/foo.git
The Debian packaging lives at:
git://git.example.com/foo-debian.git
These are meant to be stacked:
git clone git://git.example.com/foo.git
cd foo
git clone git://git.example.com/foo-debian.git
This actuall does work with ick right now, but only for public repositories
that can be accessed without authentication.
Ick has a "git" action that can be used when using the ssh protocol, which
does require authentication. This uses the worker's ssh key, but only
supports one URL:
action: git
where: host
This uses the project parmaeters git_url, git_dir, and git_ref:
parameters:
git_url: ssh://git@git.example.com/foo.git
git_ref: master
git_dir: foo
The above parametgers would tell the git action to clone the foo.git
repository to the foo directory in the workspace (checking out the master
branch). This works, but only for one repository. Daniel needs at least
two (or would need, if his repositories weren't public).
I'd like to support this for any number of repositories. I suggest the
following:
parameters:
git:
- url: ssh://git@git.example.com/foo.git
ref: master
dir: foo
- url: ssh://git@git.example.com/foo-debian.git
ref: master
dir: foo/debian
The above would do the same things as the two git clones above.
What do you all think? Especially you, Daniel? Would that be an acceptable
design (at least for now)?
9:45 a.m.
On Wed, Jun 20, 2018 at 15:30:49 +0300, Lars Wirzenius wrote:
I'd like to support this for any number of repositories. I
suggest the
following:
parameters:
git:
- url: ssh://git@git.example.com/foo.git
ref: master
dir: foo
- url: ssh://git@git.example.com/foo-debian.git
ref: master
dir: foo/debian
The above would do the same things as the two git clones above.
What do you all think? Especially you, Daniel? Would that be an acceptable
design (at least for now)?
So my main concern with this is that it remains a problem for the case of
genericisation. I currently have a single pipeline step which uses a parameter
to configure the git repository to pull from.
What I'd like would be something akin to Ansible's use of jinja2 to do:
parameters:
git:
- url: "ssh://git@server/{{ source_name }}.git"
ref: master
dir: "{{ source_name }}"
- url: "ssh://git@server/{{ source_name }}/debian.git"
ref: master
dir: "{{ source_name }}/debian"
Failing that kind of capability, your suggestion would at least mean I could move
the clones out of the shell steps I suppose :-)
D.
--
Daniel Silverstone http://www.digital-scurf.org/
PGP mail accepted and encouraged. Key Id: 3CCE BABE 206C 3B69
11 a.m.
On Wed, Jun 20, 2018 at 02:45:44PM +0100, Daniel Silverstone wrote:
On Wed, Jun 20, 2018 at 15:30:49 +0300, Lars Wirzenius wrote:
> I'd like to support this for any number of repositories. I suggest the
> following:
>
> parameters:
> git:
> - url: ssh://git@git.example.com/foo.git
> ref: master
> dir: foo
> - url: ssh://git@git.example.com/foo-debian.git
> ref: master
> dir: foo/debian
>
> The above would do the same things as the two git clones above.
>
> What do you all think? Especially you, Daniel? Would that be an acceptable
> design (at least for now)?
So my main concern with this is that it remains a problem for the case of
genericisation. I currently have a single pipeline step which uses a parameter
to configure the git repository to pull from.
What I'd like would be something akin to Ansible's use of jinja2 to do:
parameters:
git:
- url: "ssh://git@server/{{ source_name }}.git"
ref: master
dir: "{{ source_name }}"
- url: "ssh://git@server/{{ source_name }}/debian.git"
ref: master
dir: "{{ source_name }}/debian"
Failing that kind of capability, your suggestion would at least mean I could move
the clones out of the shell steps I suppose :-)
Hm. Supporting something like jinja2 in the controller is something
that may become necessary later on, but it's a complication that I'd
like to avoid until it is necessary to have it. Even if that means
more verbose .ick file. I guess it'd be possible to only support it
icktool, but I'll have to think about this.
Is your example really saving much typing? Since parameters are per
project, the following stanza would need to be repeated for each
project:
parameters:
git:
- url: "ssh://git@server/{{ source_name }}.git"
ref: master
dir: "{{ source_name }}"
- url: "ssh://git@server/{{ source_name }}/debian.git"
ref: master
dir: "{{ source_name }}/debian"
I recognize that the .ick files get rather verbose as they are
currently defined. For the immediate future, I prefer that over a more
complicated controller. Later on, as we (as the ick project) gain more
experience of what the pain points are, we can make the .ick language
more powerful.
Maybe we can make the .ick language more powerful with better
defaults? Perhaps by having a "default parameters for all projects"
section? Also, sets of pipelines for different styles of projects?
defaults:
git_defaults:
url_base: ssh://git@git.example.com/
ref: master
dir: src
default_pipelines:
kinnison_style:
- get_source_from_git
- build_lua_project
- build_deb
projects:
- project: foo
style: kinnison_style
parameters:
git:
- repo: foo.git
- repo: foo-debian.git
dir: src/debian
- project: bar
style: kinnison_style
parameters:
git:
- url: ssh://git@other.git.example.com/bar.git
- repo: bar-debian.git
dir: src/debian
This would build two projects, foo and bar. The git action would look
at the git_defaults parameter, and append the repo to url_base, unless
the full url was given. Thus the above would be equivalent of:
projects:
- project: foo
parameters:
git:
- url: ssh://git@git.example.com/foo.git
ref: master
dir: src
- url: ssh://git@git.example.com/foo-debian.git
ref: master
dir: src/debian
pipelines:
- get_source_from_git
- build_kinnison_project
- build_deb
- project: bar
parameters:
git:
- url: ssh://git@other.git.example.com/bar.git
ref: master
dir: src
- url: ssh://git@git.example.com/bar-debian.git
ref: master
dir: src/debian
pipelines:
- get_source_from_git
- build_kinnison_project
- build_deb
No templating complexity, but rather less repetition. It's not as
generic a solution, of course. What do you think?
--
I want to build worthwhile things that might last. --joeyh
11:06 a.m.
On Wed, Jun 20, 2018 at 18:00:43 +0300, Lars Wirzenius wrote:
Hm. Supporting something like jinja2 in the controller is something
that may become necessary later on, but it's a complication that I'd
like to avoid until it is necessary to have it. Even if that means
more verbose .ick file. I guess it'd be possible to only support it
icktool, but I'll have to think about this.
Supported in icktool and then uploading specialised pipelines would be
acceptable.
Is your example really saving much typing? Since parameters are per
project, the following stanza would need to be repeated for each
project:
doh, I meant for that to be part of a git pipeline step, and for source_name to
be one of the parameters set in the project. I was not explaining myself well.
I recognize that the .ick files get rather verbose as they are
currently defined. For the immediate future, I prefer that over a more
complicated controller. Later on, as we (as the ick project) gain more
experience of what the pain points are, we can make the .ick language
more powerful.
It's possible that perhaps Yaml anchors and dictionary merges might help with
some of this.
Maybe we can make the .ick language more powerful with better
defaults? Perhaps by having a "default parameters for all projects"
section? Also, sets of pipelines for different styles of projects?
[snip]
No templating complexity, but rather less repetition. It's not
as
generic a solution, of course. What do you think?
I think that introduces a little bit too much magic into the git action,
and another keyword (style) which I'm not convinced by. For now let's
stick as we are, and we can always consider preprocessing with jinja2
in icktool or similar at a later date.
At worst, for now, I have a shell script generate my ick file from a bunch
of snippets and running sed :-)
D.
--
Daniel Silverstone http://www.digital-scurf.org/
PGP mail accepted and encouraged. Key Id: 3CCE BABE 206C 3B69
4:54 a.m.
On Wed, 2018-06-20 at 16:06 +0100, Daniel Silverstone wrote:
I think that introduces a little bit too much magic into the git
action,
and another keyword (style) which I'm not convinced by. For now let's
stick as we are, and we can always consider preprocessing with jinja2
in icktool or similar at a later date.
I'm inclined to agree. We'll revisit this later.
1812
days inactive
1813
days old
4 comments
2 participants
participants (2)
-
Daniel Silverstone -
Lars Wirzenius