Feature Request: Clean unlocked packages

[readeral]

Hi Nicolas,

I'm slowly working my way through manually cleaning out items that I don't want to commit to source control, the list of manual metadata cleaning has been helpful thankyou (although I do have some feedback on that too!) but I'm coming unstuck with unlocked packages, specifically, a selection of packages from unofficialsf. These particular packages don't have a namespace so can't be targeted using sfdx hardis:project:clean:manageditems.

What I can do is run sf retrieve start --package-name FlowScreenComponentsBasePack which will give me all the metadata in its own root folder in my project, and then I can (for now) manually pick through my org metadata and delete the equivalent files I find that match the ones in the newly pulled package root.

So, here's my idea. Given you already have a duplicate finding command, would it be reasonable to develop an "unlocked packages dedupe" command/workflow to clean up unlocked packages?

I know there's risks of false positives given there's no namespace to check against, as well as probably any number of other hazards with damaging one's own unlocked packages, so it would probably have to be a command which:

• only runs with a --package-name flag on a package-by-package basis
• reports where it finds triplicates for manual intervention
• offers a flag for report only
• offers a flag approving every file deletion step by step

I dunno, does this sound crazy? I mustn't be the only one who uses unofficialsf or other similar unconventional packages that can't be namespaced.

I'd be happy to draft up the feature if you're interested in it? I'm a little rusty with typescript, but would be able to give it a good shot.

[readeral]

sfdx hardis:org:retrieve:packageconfig -u YOUR_PROD_ORG_USER could even warn of packages without namespaces that might need to do this process.

[nvuillam]

@readeral your idea seems very interesting :)
I didn't find a way to get some package.xml from an installed package id, but I found an interesting tooling api object that seems to match your requirement ^^

https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/tooling_api_objects_package2member.htm

You could create such command probably by doing the following:

• call existing sfdx-hardis method to list installed packages
• prompt the user to select one package
• use select package subscriber pckg id to list the related sources using package2member
• dynamically generate a filterConfig.json from the tooling api results
• apply cleaning with this dynamic filterConfig.json :)

I will gladly validate such pull request and provide you support if necessary :)

You can start here ^^ https://sfdx-hardis.cloudity.com/#contributing

[readeral]

So, I'm a SOQL amateur at best, but I'm really struggling to get any usable data from package2member. I can get an array of Package2Member components, and on each component I can get the SubjectID which seems(?) to be the id for that package component, but I'm not exactly sure how I can use that id to get metadata names etc. using the tooling API.

[readeral]

Oof this isn't a small job.

I've managed to get all the metadata from an unlocked package through only the tooling API, but it requires a string of calls that may add up quickly depending on the size...

Given you're far more experienced than me, maybe you can spot if in this process a composite request or group operation can be used.

Each package member needs to be queried on the sobject endpoint to get enough usable data for this feature. Sadly, Package2Member does not provide the SObjectName of each of its members, only the prefix, so this needs to be determined before each item can be retrieved.

I don't have a minimal code example yet as I've only been playing around in Postman Flow editor, but my current process is:

• hit /tooling/sobjects to list the Tooling API Objects, store the SObjectName and KeyPrefix as keyvalue pairs in an object variable api_name.
• querying Package2Member: /tooling/query/?q=SELECT SubjectID, SubjectKeyPrefix FROM Package2Member WHERE SubscriberPackageId='{SubscriberPackageId}' so we have a list of package element IDs and the SObject prefix
• iterative API calls to tooling/sobjects/{SObjectName}/{SubjectID} looking up SObjectName from api_name using the SubjectKeyPrefix, returning the relevant metadata for that component ID.

Doing a little transformation, hitting the tooling/sobjects/{SObjectName}/ endpoint can grouped per SObjectName, which in my small test would reduce API calls to that endpoint from 41 to 10.

Once we have the data, building up the filterConfig.json isn't entirely straightforward.

Package members don't all follow the same schema for names depending on their SObject:

• a member must have a name OR developerName
• a member can have a fullName

fullName almost does the job of identifying every component in the project, the exceptions (in my limited dataset) were CustomObject, PermissionSet and ExternalString (not individual files but found within the CustomLabels.labels-meta.xml, ironically within the tag).

[nvuillam]

@readeral I'm no expert of tooling api at all, but your draft seems to go in the good direction :)

If you succeeded to go down to 10 requests, it's totally acceptable because it's not a script that would be called 20 times a day ^^

[readeral]

Humph. Despite being in their documentation it seems the composite endpoint doesn't actually exist on the tooling api. Neither the listed endpoint in these docs, nor the doc version from Summer 22 (which actually includes '/tooling/' in it's url) work. So that reduction in requests isn't gonna happen unless there's another way for me to do a bulk query.

I've done some basic work on the command. I'll commit what I have for you to have a look at, so you can tell me to change what I'm doing so far! I've changed apiUtils to perform a global describe API call (conn.tooling.describeGlobal), I will also need to look at adding conn.tooling.retrieve as well for the final roundup of required data, but if the composite isn't available, then maybe that can all be done via a query anyhow.

[readeral]

See branch here: https://github.com/readeral/sfdx-hardis/tree/unlocked-packages

[nvuillam]

@readeral please create a PR with draft statut, and i'll have a look :)

[readeral]

#769