What if we defined a story point to be:
"The relative likelihood that this whole Scrum team is inadvertently, collectively under the wrong impression about what it will actually take to make the product do something that helps the user with the problem under consideration."
Things it could take which cannot be foreknown include: time, effort, policy changes to the org, political consensus, freedom to try crazy ideas, beginner's luck, unlearning biases, new tools, a no-fly-zone, a clear splash zone, 3 outages, coinciding calendars, a partridge, a pizza tree, a towel, chewing gum, foil, duct tape & 1 thin mint.
Said differently again: The less likely that the Scrum team believes they are to mis-intuit what needs to happen to deliver User Story B (relative to User Story X, which had greater Story points just before the same team pulled it into the Sprint to work on X) the LOWER the User Story point. The Scrum team is evaluating its own degree of fallibility inside the context of producing User Story B amidst current conditions and team members, everything else being equal in the enterprise.
The Scrum team also assumes in the assessment, the enterprise will support and empower at historic levels, the Scrum team to do whatever it takes to get User Story B "Done, by the team's definition of Done." So, as the rest of the enterprise is even more supportive and empowering of the Scrum Team, it follows that the User Story points the Scrum Team assigns to user stories decreases, on average, or they keep getting Done early and have the option to get more unplanned stories Done by the end of the Sprint, or do something even more valuable with the slack time. Good news!
And there's catch. Isn't there always, though? In the spirit of increasing quality, expandable, maintainable architecture, and clean code, the team's definition of Done, grows only more stringent over time, slowly raising the bar on technical excellence and good design which the Scrum team(s) hold themselves to. That'll really take something to achieve each Sprint, and increases User Story Points.
Now, one more wrinkle: when the rest of the enterprise is less supportive and empowering of the Scrum Team, it then follows that the User Story points which the Scrum Team assigns to user stories increases, on average to reflect impairment. If the Scrum Team neglects baking those ambient variances into their Story Point estimates, a common pattern emerges wherein they keep not getting Done each Sprint.
A chain is only as strong as its weakest link. A dependency chain runs only as fast as it's slowest link. So, either the powers that be in the enterprise respond to the Scrum Team's requests (for decisions, unrestricted access, or sensitive information) faster than the Scrum Team is moving already, or better still: authorize the Scrum Team to act on your behalf, so that you (the executive leadership) or some committee is not the bottleneck, slowing down the chain of events (dependencies) that must happen for delivery to occur within the Sprint time box.
Given this is how the whole system of Scrum in the enterprise works, one can see how self-sabotaging it would be for people outside the Scrum team to tamper with User Story Points, trying to sway them or stonewall the Scrum team from facts about the rest of the enterprise, or vice versa. Transparency itself, in both directions informs the Scrum team which prevents false impressions, effectively LOWERING User Story points they assign to any given User Story, so MORE gets Done each Sprint. Let's accept that even an all-knowing Scrum Team or Team of teams, still works in concert with its corporate environment. So, how the rest of the enterprise functions or not, impacts the outcome for better or worse.
Always assume that transparency is lacking in your system until you have evidence to the contrary. Decision latency of less than 1 day is extremely rare in multi-leveled corporate structures with many hand-offs and reporting lines. You cannot improve what you do not measure, and it runs against human nature to complain to higher-ups about their slowness to decide, let alone publish their short-comings. Let's exercise our personal leadership. Leaders go first here. Tattle on yourself, and improve. You'll thank yourself in the future.