Skip to main content

User based queue mapping for Capacity Scheduler

When I  started to use Capacity Scheduler hierarchical queue features on top of Hortonworks' HDP 2.0 I have immediately realized that I need automatic assignment of job to queue based on username.

Sounds easy and useful? Yes! But could not find any configuration parameter and example for that.

I found only references to use mapred.job.queuename config option. This can be configured in HIVE via set mapred.job.queuename=yourqueue or using -Dmapred.job.queuename=yourqueue as a hadoop command argument.

After some hours of unavailing googling I have checked the corresponding code part and have been shocked. This is available only since HADOOP-2.6 (HDP-2.2). Check YARN-2411 for details. According to the CHANGELOG this is a relatively new feature. So sadly this is not available to me until an upgrade.

:(

See below an example based on YARN-2411 to use it in Hadoop 2.6 or higher for Hortonworks HDP-2.2

1. user1 is mapped to queue1, group1 is mapped to queue2:

yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.queue-mappings=u:user1:queue1,g:group1:queue2

2. To map users to queues with the same name as the user:

yarn.scheduler.capacity.queue-mappings-override.enable=true
yarn.scheduler.capacity.queue-mappings=u:%user:%user


Update: As a workaround I have configured the default queue to have very low 10% capacity with the possibility to use even 100% and created another queue which has 90% capacity (up to 100%) and can be used only by a special user. Also enabled ProportionalCapacityPreemptionPolicy so the system will kill over allocated resources from default queue in case prio queue needs those.

Example config:
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill=20000
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval=10000
yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round=0.1
yarn.resourcemanager.scheduler.monitor.enable=true
yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.acl_administer_jobs=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.acl_submit_jobs=*
yarn.scheduler.capacity.root.default.capacity=10
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=10
yarn.scheduler.capacity.root.prio.acl_submit_applications=specialuser
yarn.scheduler.capacity.root.prio.acl_submit_jobs=specialuser
yarn.scheduler.capacity.root.prio.capacity=90
yarn.scheduler.capacity.root.prio.maximum-capacity=100
yarn.scheduler.capacity.root.prio.state=RUNNING
yarn.scheduler.capacity.root.prio.user-limit-factor=10
yarn.scheduler.capacity.root.queues=default,prio

Comments

Popular posts from this blog

Insufficient Disk Space reported under wine

Did you try to install/setup any Windows Application - actually a Game what else could be necessary - and got a message that you do not have enough free space on your drive meanwhile you had lot of free space on the chosen mounted partition? You will learn the problem and hopefully the solution too. (Of course I suppose it is not the real situation you have no enough space. If so do not read ahead.) The problem is that wine does not check the amount of free space on the mounted partition corresponds to the selected directory but reports the free on the root of the directory the partition mounted to . ;( Probably it is not clean so here is an example: Let say you have / only and something is mounted as /mnt/part1 If you directly select /mnt/part1 during installation wine will check free space in fact on / and does not calculate free on the partition mounted under /mnt/part1. How to solve it you may ask? It is easy. Start winecfg and create a new drive with the directory you want to use.

Ansible: Using multiple tags and untagged tag together

I have lots of Ansible playbooks with many roles in each. However when you are installing different minor version of the same software stack, there are only minor differences between the steps. In this case it does not make much sense to "copy paste" the whole role so I just wanted to use tags. I wanted to use untagged tasks as common tasks and tagged tasks for version specific tasks. To make it clear here is an example. If you have a long os related role which does ssh config, web config, database install and creation and many more but sometimes you need java-6 or java-7 it is easy to add task and tag those according to this. Than my theory was that I can run ansible-playbook --tags=untagged,java6 to install the stack with java6 and ansible-playbook --tags=untagged,java7 to install same stack with java7. However this does not work. I have checked the Ansible source code and found why it is not working. Since I was not sure if this is a bug or by design I have opened a

Python Azure ML SDK issue on Ubuntu 22.04

It has been quite a while since I posted last time. Why? Because simply I did not run into any issue worth to share. But now! I did.  Recently we are doing some Machine Learning on Azure using Azure Machine Learning Python SDK. No problem you might think. Well. As it turned out Ubuntu 22.04 is not supported. And this is clearly said in a message. Which is in fact a lie. The Error message: NotImplementedError: Linux distribution ubuntu 22.04 does not have automatic support. Missing packages: {'liblttng-ust.so.0'} .NET Core 3.1 can still be used via `dotnetcore2` if the required dependencies are installed. Visit https://aka.ms/dotnet-install-linux for Linux distro specific .NET Core install instructions. Follow your distro specific instructions to install `dotnet-runtime-*` and replace `*` with `3.1.23`. Ok but what is this? And why? So as the error mentions dotnetcore2==3.1.23 Python package uses .NET Core 3.1 but Ubuntu 22.04 has only dotnet6 packages. And also Micro