Give a detail readme for server

[starsareintherose]

please give details for the server, or give a shell for each script.

For example, how to archive

r-spp
r-propr
r-catnet
r-pkgmaker

[sukanka]

TLDR

For usage, run with argument -h.

Detailed steps

0. copy all my py files to Packages dir.

1. update database
   download the database from release, and move it to /tmp/dbmanager/biometadata.db, run this command to update database.

./dbmanager.py  -m "3.15" --db /tmp/dbmanager/biometadata.db

2. prepare the pkg list to be archived, say, a file named pkglist.txt,

r-spp
r-propr
r-catnet
r-pkgmaker

3. run depends_updater -a means auto archive

./depends_updater.py -a -f pkglist.txt -db /tmp/dbmanager/biometadata.db -o out.txt

4. Check out.txt, you may need to add these pkgs to BioArch Repo

[starsareintherose]

what about archive? It's not safe to under Packages directory, better to run under Tools directory

[sukanka]

what about archive? It's not safe to under Packages directory, better to run under Tools directory

archive is merged into depends_updater. Run it in Package dir, and check the modified files if you want. or you can use other argument to specify the path to Package dir,
see

➜  Packages git:(master) ./depends_updater.py -h
usage: depends_updater.py [-h] [-f FILE] [-p BIOARCH_PATH] [-db DB] [--bioc_min_ver BIOC_MIN_VER] [--bioc_meta_mirror BIOC_META_MIRROR] [-o OUTPUT] [-a]

update the depends of R packages from CRAN and Bioconductor automatically

options:
  -h, --help            show this help message and exit
  -f FILE, --file FILE  The file that contains the pkgname to be archived, one pkgname per line (default: None)
  -p BIOARCH_PATH, --bioarch_path BIOARCH_PATH
                        The path of BioArchLinux repo (default: BioArchLinux)
  -db DB                The database file used to query metadata of packages (default: /tmp/dbmanager/sqlite.db)
  --bioc_min_ver BIOC_MIN_VER
                        The minimum version of Bioconductor supported, must be greater than 3.0 (default: 3.0)
  --bioc_meta_mirror BIOC_META_MIRROR
                        The server used to get all version numbers of BIOC (default: https://bioconductor.org)
  -o OUTPUT, --output OUTPUT
                        The file to save newly added depends name (default: added_depends.txt)
  -a, --auto-archive    Automatically archive pkgs that are not in CRAN or the latest BIOC release (default: False)

[starsareintherose]

./depends_updater.py -a -f list -db biometadata3.15.db  -o out.list -p ../../Packages/BioArchLinux/
INFO:root:Updating r-cvms
INFO:root:Writing new PKGBUILD for cvms
Traceback (most recent call last):
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 385, in <module>
    update_depends_by_file(args.file, args.bioarch_path, args.db,
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 349, in update_depends_by_file
    if auto_archive and pkginfo.is_archived():
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 140, in is_archived
    if not self.desc:  # not in database, archived in CRAN
AttributeError: 'PkgInfo' object has no attribute 'desc'

could you check it on server

[sukanka]

./depends_updater.py -a -f list -db biometadata3.15.db  -o out.list -p ../../Packages/BioArchLinux/
INFO:root:Updating r-cvms
INFO:root:Writing new PKGBUILD for cvms
Traceback (most recent call last):
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 385, in <module>
    update_depends_by_file(args.file, args.bioarch_path, args.db,
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 349, in update_depends_by_file
    if auto_archive and pkginfo.is_archived():
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 140, in is_archived
    if not self.desc:  # not in database, archived in CRAN
AttributeError: 'PkgInfo' object has no attribute 'desc'

could you check it on server

did you update the database first? biometadata3.15.db should contain all packages from BIOC 1.8 to BIOC 3.16 and pkgs from CRAN

[starsareintherose]

Of course, I update it.

[sukanka]

r-cvms

OK, confirmed, this function is buggy, please do not use -a

[sukanka]

./depends_updater.py -a -f list -db biometadata3.15.db  -o out.list -p ../../Packages/BioArchLinux/
INFO:root:Updating r-cvms
INFO:root:Writing new PKGBUILD for cvms
Traceback (most recent call last):
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 385, in <module>
    update_depends_by_file(args.file, args.bioarch_path, args.db,
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 349, in update_depends_by_file
    if auto_archive and pkginfo.is_archived():
  File "/usr/share/lilac/Tools/updater/./depends_updater.py", line 140, in is_archived
    if not self.desc:  # not in database, archived in CRAN
AttributeError: 'PkgInfo' object has no attribute 'desc'

could you check it on server

fixed in 9726e5e
please try again

I have tested it, it works, but DO have a look before you push it to the master branch

[starsareintherose]

On the server is okay, but local fails

/depends_updater.py -a -f list -db biometadata3.15.db  -o out.list -p ../../Packages/BioArchLinux/
Traceback (most recent call last):
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/./depends_updater.py", line 438, in <module>
    update_depends_by_file(args.file, args.bioarch_path, args.db,
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/./depends_updater.py", line 363, in update_depends_by_file
    bioc_versions = get_bioc_versions(bioc_meta_mirror)
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/dbmanager.py", line 96, in get_bioc_versions
    version_page = requests.get(
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 456, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 352, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 217, in proxy_manager_for
    manager = self.proxy_manager[proxy] = SOCKSProxyManager(
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 62, in SOCKSProxyManager
    raise InvalidSchema("Missing dependencies for SOCKS support.")
requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support.

[starsareintherose]

Another thing is that how can you know if a pkg exist in which version db

[lilydjwg]

Install python-pysocks for the exception.

[starsareintherose]

still, I am not sure if the proxy influence

./depends_updater.py -a -f list -db biometadata3.15.db  -o out.list -p ../../Packages/BioArchLinux/
Traceback (most recent call last):
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/./depends_updater.py", line 438, in <module>
    update_depends_by_file(args.file, args.bioarch_path, args.db,
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/./depends_updater.py", line 363, in update_depends_by_file
    bioc_versions = get_bioc_versions(bioc_meta_mirror)
  File "/home/guoyi/Downloads/bioarchlinux-tools/updater/dbmanager.py", line 96, in get_bioc_versions
    version_page = requests.get(
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 456, in send
    conn = self.get_connection(request.url, proxies)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 352, in get_connection
    proxy_manager = self.proxy_manager_for(proxy)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 217, in proxy_manager_for
    manager = self.proxy_manager[proxy] = SOCKSProxyManager(
  File "/usr/lib/python3.10/site-packages/urllib3/contrib/socks.py", line 198, in __init__
    raise ValueError("Unable to determine SOCKS version from %s" % proxy_url)
ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/

[starsareintherose]

@lilydjwg Could convert these scripts into pre_build function and then I can use it directly by adding function to lilac.yaml?

[lilydjwg]

Use socks5://?

Yes if I can figure out how to use them.